Thom Hogan

As many of you know, I have a software developer background that dates back to 1976, including a number of sophisticated things that have been built into today's modern OS's (e.g. Ram Doubler's memory encryption trick is now adopted by macOS; Microsoft bought Virtual from my company). Along the way I've created a number of software products that needed to exist, but at the time weren't available in any form (you have me to thank for true table making features as well as linking spell and grammar checks into word processors, for instance). 

If you know my Silicon Valley background, it's probably been surprising to you that I haven't yet created (or helped create) any software product for digital cameras, particularly the Nikon ones that I spend most of my time with. 

Frankly, the software business is treacherous, and I was glad to leave it behind after the turn of the century. Still, as you've probably noted I've been critical of a lot of digital software features, including those built into cameras. A couple of times recently I've tinkered with ideas with some of my developer friends, including an unreleased monochrome-only raw converter program we called RawTriX (pronounced raw tricks). I also often suggest (and test) things for my friends who are still putting out products of their own. 

Well, today I get back in the software game. 

In a big and highly unexpected way.

Ever since the introduction of the Nikon Z9 I've been digging into the physical aspects of that camera as I tried to understand what Nikon had actually designed, and how it was different. The quad-core nature of the EXPEED7 processor seemed underutilized, and as we've seen from subsequent firmware updates, the original firmware left a lot off the table. 

My work on the Z9 firmware included disassembling and using customized test equipment to monitor what was happening on the main digital board within the camera, and in particular, how the EXPEED7 processor was utilized. That led me to a couple of surprising revelations. It turns out that there is a way to pass new processes into the ARM processors within the EXPEED7 chip and have them execute; the trick was to not step on what Nikon was doing. However, I found that I could easily piggy-back on what Nikon did. As it turns out, I couldn't decipher all of Nikon's "black box" API (Application Programming Interface), but I did crack enough to be able to get the EXPEED7 chip to execute code I sent to it, as well as get access to the menus and information advisories.

Once I figured the control bits out, the real issue became getting past the encryption and verification procedures Nikon uses in creating their firmware files and the individual modules within them. As others before me have discovered, you not only have to be able to understand the instructions and data registers used in the various computing components—there's more than just the ARM cores in an EXPEED7 chip—but you also have to figure out how to add your code into the camera's firmware without the camera's built-in self-protection systems kicking in and bricking the camera, and also not stepping on privileged processes as you do it. Moreover, you also have to hook into the menu and control response systems, though as it turns out, Nikon left those hooks exposed.

One thing that I was able to isolate and understand early was how the Voice Memo system got invoked and interacted with the other hardware in the camera (e.g. microphone and speaker). This led me to one of those Aha! moments: back in the early 90's a friend shared with me a small speech recognition process he created (which was being tested on the old EO 440 PenPoint machines I helped design and evangelized, which were also ARM-based). What if I could stick that routine into the Voice Memo hooks? 

And thus was born Hey Nikki.

Yep, just like Apple's Hey Siri, only you say Nikki (neek-ee in Japan ;~) to invoke the system. With my mods in place, you will now see a new Voice memo option of Hey Nikki appended to the voice options menu. It uses the simple Z9 toggle on/off construct.

When you use Hey Nikki via a button programmed for Voice memo, this is what you'll initially see:

...and after a brief moment, you'll briefly see the following message:

I'm still trying to figure out a better way of doing this. There's a brief delay between the messages as I try to get my displays into Nikon's usual sequences, and the final completion message is actually happening a fair amount of time after the change has already been made. I also use a generic error message should Nikki not understand what you said, and I'd like to make that more specific in the future. Also in the current implementation it's possible to press the shutter release before the second message appears, and it will be indefinite as to whether the change was actually made or not: too close to the original message and it wasn't, but closer to the second message and it was, but you won't know which is which at the moment. Yucky. I'm working on another approach that should fix this.

So, starting today, Just talk, say the magic words, and your Z9 will change its settings! At least the ones I've deciphered and added so far.

This first version of Hey Nikki will probably seem a little strange and toy-like. That's because I'm still trying to figure out what things I can control and which things I know the Nikon API for. Here are the very first words that Hey Nikki understands:

• Hey Nikki 15 frames (or 10, 12, 20) — sets the Continuous High frame rates (camera must be set to CH to work, otherwise in the current implementation nothing happens; I'll work on that ;~)
• Hey Nikki Luxos — turns the AF Assist Lamp on or off ;~) (Yes, a Harry Potter reference. Imagine your child learning that your camera understands wizardry!)
• Hey Nikki mark 5 (or delete, protect, mark 1, 2, 3, or 4) — tags the currently displayed image with the desired rating; if in composing mode, nothing happens (at the moment Nikki doesn't provide feedback, but I'll work on that, too)
• Hey Nikki send — marks the currently displayed image for Send to smart device
• Hey Nikki Aperture (or Manual, Shutter, or Program) — sets the Exposure mode (MODE button equivalent)
• Hey Nikki SHOOTING A (or B, C, D, or CUSTOM A, B, C, D) — changes the camera to the named SHOOTING menu bank (or CUSTOM SETTING bank) This was the big win on this first iteration of Hey Nikki: you don't have to dedicate a button to banks; I'm now working on allowing the combo you can't do with buttons, e.g. Hey Nikki SHOOTING A CUSTOM A

(Full documentation comes in a Read Me file with the installer. You'll need a copy of Nikon's C5.00 firmware to create the full Hey Nikki install binary.)

Yeah, not a lot of meat on the bone in this first version. But as I discover and understand more of Nikon's API and add that to my code base, I'll be updating the program. Uh, firmware. At least until we hit the second full version of Hey Nikki, updates will be free.

Right. That brings me as to how you load Hey Nikki into the camera. 

I'm piggybacking my code onto a copy of Nikon's current firmware binary (fortunately, there's plenty of space in the camera for my routines, at least so far). Which means that you get it in the camera by installing the Z_9_0511.BIN file you create from my installer (I'm using 5.11 as the version number because Nikon might update firmware files again for the Z9, and I don't want to use a number they're likely to use soon). Unfortunately, the camera only recognizes Nikon's own firmware file name specification, so I can't rename the file something like Z9_HeyNikki_0511.BIN.

The drawback to the firmware installation is that if Nikon updates the Z9 firmware, you'll have to wait for me to convert Hey Nikki to the new firmware binary. (Interestingly, you can install a new Nikon firmware update over mine, which I was just able to test with several of the recent Nikon updates, but it will remove Hey Nikki.)

As many of you know, one of the my long-standing requests for digital cameras has been programmability. I'm still trying to figure out Nikon's Z9 Menu Settings file (NCSET010.BIN [sic]). I have hope that I can fully decipher that file soon so I can create a computer-editable version that Hey Nikki can then trick the camera into loading instead (e.g. Hey Nikki load BIF). But the other thing that would be interesting is to string multiple commands together (e.g. Hey Nikki Lossless compressed Auto ISO Incandescent), or to set all the options for one of the more complex methods (e.g. Interval timer shooting, Focus shift shooting, or Auto capture) without dropping into the menus. Yeah, this sort of extension would be geeky, but I'm looking for rapid camera reconfiguration methods, not simplicity. So Nikki is a geekess. Don’t expect her to chat with you (that would be a possible future software product, NikkiChat ;~).

Okay, by this point you're either in for some geekdom or not (hey, you bought a Z9, which has hundreds of geeky settings to start with!). Price for Hey Nikki is for a limited time just US$19.99 for all 1.x versions I come out with (long term the price needs to be higher to support all the iteration work and support that needs to be done, so get it at this price while you can). 

If you'd like to purchase the 1.0 version of Hey Nikki for your Z9, use the following link to start the purchase process: