I recently made the decision to get a new TV, but couldn’t settle on which size to get. One Saturday morning I came up with the idea to use Apple’s new ARKit tech to help solve this problem, and shortly after TV Plan was born.

In Summer 2017, Apple made it offical that they were getting serious about Augmented Reality. After seeing what the tech was capable of, I realized immediately this was something I wanted to get involved in. Since June, I’ve been learning and tinkering with ARKit to see what it’s capable of, but only recently has an obvious real-world use case come to mind.

An important opportunity I’ve identified has been for simple utilities which solve problems that couldn’t otherwise be addressed without AR. My problem was “will a 55 inch TV look too big in my living room?”.

A lesson learned from this project was the effectiveness of lighting and shadows. Adding a soft shadow behind the TV resulted in geometry looking way less like a superimposed 3D mesh slapped on top of my camera’s feed, and more like something which was actually “there”. Although subtle and not immediately obvious, I expect improvements in light estimation to be critical to the success of AR in the future.

Here’s an example of the TV with and without shadows:

As for next steps, I’ve been collecting feedback online and offline, and will be cycling quick wins into the app in the near future. For example, a common suggestion has been to add support for different TV sizes. Turns out my assumption that TVs max out around 75” was pretty naive 🤓.

Since getting an Apple Watch last fall, I’ve been disappointed by the lack of content. To help address this, I made my own game a few months ago (a 3D RPG), but obviously it still didn’t address the bigger issue. An idea I had was to port an existing catalog, and emulation made perfect sense.

The result is a surprisingly usable emulator which I’m calling Giovanni after the super-villain from my favourite Game Boy game, Pokemon Yellow. Ironically, I’ve only ever played the game on an emulator, as growing up I didn’t have access to the real deal. In a way, I feel this is my way of giving back to the community.

That feeling from accomplishing something you thought was too crazy to actually work is phenomenal. Here’s what Pokemon Yellow looks like on a Series 2 Apple watch:

One of the big challenges was to find the right balance between framerate and performance. As you can see, it’s a bit sluggish and unresponsive, but as a prototype, I think it answers the question of “is this possible”.

A Few Lessons Learned

For the sake of prototyping, I committed to doing as little unnecessary work as possible, so extending an existing emulator was my first approach. However as platform limitations and ubiquities surfaced, I discovered it was way less effort to just start from scratch and only pull in what I needed from open source projects.

My go-to emulator was Provenance, an open-source front-end for iOS. Having made contributions (albeit minor ones) in the past, I was semi-familiar with the codebase. My first approach was to have the watchOS extension ask iOS for a list of games names, then when a game was selected, download it and run locally. The first issue was with WatchConnectivity and Realm. Because Realm requires queries to be made on the main thread, that meant I wouldn’t be able to communicate with iOS unless the app was running, which wouldn’t be an ideal user experience. After investigating having querying Realm without WatchConnectivity and not having much luck, I realized Provenance offered a level of sophistication that wasn’t really necessary at this point.

Taking a step back, I realized Provenance (and others) are just thin layers on top of Gambatte —— where all the actual emulation was being done. After cloning the repo and looking at example code, I realized Gambatte was already extremely high-level, already offering support for loading ROMs, saving/loading, even GameShark.

Graphics on watchOS

Oh yeah, up until this point I was working under the fatal assumption that watchOS supported OpenGL or Metal (because of SceneKit). As it turns out, it doesn’t. I imagine if I did this investigation beforehand, it probably would’ve deterred me from even beginning. Nonetheless, I was too far in.

Recalling a Hacker News post about how some guys from Facebook got Doom running on watchOS not too long before, I discovered they were rendering to a UIImage using Core Graphics. Looking at what Gambatte was filling it’s video buffer with revealed that it too was just outputting pixel data. 

I realized I didn’t know enough about feeding pixel data into a Core Graphics Context, so I created a Swift Playground to play around with creating images from a pixel buffer. As someone who learns best through trial and error (“what does changing this thing do?”), the ability to get immediate visual feedback was amazing.

Immediately after this exercise, I was finally able to get something on-screen, but the colors were all wrong. By adjusting the byte order and composition options (and with a little more trial and error), I was able to get what finally looked right:

For input, making a button on screen for every single input wasn’t desirable, so I took advantage of gestures and the Digital Crown. By allowing the user to pan on screen for directions, rotate the Digital Crown for up and down, and tap the screen for A, I was able to eliminate buttons until I was left with Select, Start, and B.

Touching the screen for movement isn’t a great interaction, but being able to use the Crown worked out a lot better than originally anticipated. Scrolling through a list of options is basically what the Crown was made for, and if the framerate was even slightly higher, the interaction could almost be better than a hardware D-pad.

I’m anticipating revisiting the project once watchOS4/next generation Apple Watches are out, just to see if there are any opportunities to get a higher framerate. Maybe Apple will even announce Metal for watchOS at WWDC this summer!

——

This project was a ton of fun to work on, and a great reminder that even though unknowns are scary, you really don’t know what’s possible without trying. The end result was way less effort and code than I originally anticipated.

I’m still deciding if it’s worth investing more time into improving performance, but for the sake of getting some feedback, I’ve open sourced everything on Github. If you’re interested in contributing feel free to submit a PR. For any thoughts or feedback, let me know on Twitter at @_gabrieloc or through email.

I’ve recently started getting into drones, and like so many others, it all started with cheap toy quadcopters.

For under $50, you can get ahold of a loud little flying piece of plastic from Amazon, and they’re a lot of fun. Some of them even come with cameras and Wi-Fi for control via a mobile app. Unfortunately, these apps are pretty low quality — they’re unreliable and frustrating to use, and look out of place in 2017. The more I used these apps, the more frustrated I got, so I started thinking about ways I could provide a better solution, and two months later I emerged with two things:

1. An iOS app for flying quadcopters called SCARAB, and

2. An open-source project for building RC apps called QuadKit

——

Continue reading this article on Medium, where it was syndicated to HackerNoon.

When I finally got around to picking up an Apple Watch in Fall 2016, one of the first things I noticed was a lack of real games.

After hearing about Doom running on watchOS earlier in the year, and Apple bringing game technologies to watchOS 3, I spotted an opportunity. Inspired by classic dungeon crawler games, I began working on what would become SAQQARA, an homage to weird DOS games from the late 90s.

While the tech was fun to play around with at first, computational limitations of the platform became apparent early on, bringing perfomance front and center during development.

With Instruments.app totally unsupported for watchOS, I had to more or less improvise with stopwatch-logging methods wrapping heavy operations to identify slow algorithms or inappropriate data types. All in all, an excellent reminder of easy it is to take for granted the (by comparison) ultra-powerful devices we keep in our pockets.

One of the bigger architectural decisions that had to be made early on was which maze-generation algorithm I would use. While prototyping gameplay early on, the first choice was depth-first search, which yielded unpredictable corridors which spanned the area of the map.

This technique was dropped not longer after, as corridors and no rooms soon became tedious in-game, and generating a map with an area greater than 5x5 units would bring the process above the system’s High Water mark, crashing it shortly after launch.

The second option was something closer to recursive division, which was not only far faster, but resulted in a good mix of big and small rooms, separated by short corridors.

For version 1.0, I was adamant on shipping essentials, and adding content iteratively in updates. After being rejected by Apple the first 4 times, SAQQARA went live right before the holiday break. As always, a great reminder to RTFM.

——

SAQQARA on the App Store

What does “Saqqara” mean?

In late 2015, I noticed a set of tweets circulating of people playing chess using Emojis in iMessage.

After trying it out for myself, I beceame immediately exhausted by the process of removing and inserting emojis manually with the Emoji keyboard, and decided it would be less effort to make a tool to do the hard stuff for me. With iOS custom keyboards emerging around that time, I took a weekend off from ALEC to throw together Emoji Chess Keyboard.

Unsure of whether Apple would approve of the concept, I hastily submitted a working prototype, which actually ended up getting approved.

Shortly after, I submitted a second version which then cleaned up some of the rough edges and added polish. Unfortunately, Apple decided they no longer liked the concept, and rejected every re-submission and appeal request I made. I’ve accepted that 1.1 will never see the light of day.

——

More Info:

Emoji Chess Keyboard on the App Store

Towards the end of 2014, I was preparing for a long trip to South Korea.

I build ALEC as a way to prepare for my trip by improving my speaking ability, as well as get my feet wet with some of the new iOS 8 technologies which were being rolled out around the time.

I ended up learning so much that I even wrote a short article on Medium, describing my goals, process, and challenges of the project. It’s about an 8 minute article, and I reccomend checking it out to get a sense of what it’s like building something while living in another country: Making ALEC, a Language Exchange Service Built While Travelling.

——

More Info:

ALEC Marketing Site

At one point in 2013, I decided I was fed up with sending the same messages to the same people when we’d hang out — “Where are you now”, “Are you close?”, “I’ll be late”, etc.

Our phones all had ways of communicating that information passively anyways, so I figured there might be an opportunity to have an app do the work. The goal was always to make something that was so frictionless to use, that there’d be a greater benefit to using it vs. just sending messages.

I defined two categories of users — “organizers” and “joiners”, and distilled the workflow for organizers down into three easy steps:

Once an organizer would create an event, invited members would all receive push notifications, which would prompt them to either accept or reject the invitation. Upon accepting the invitation, they’d see themselves and other joiners in the app immediately. Not much more to it than that.

Another aspect to events was that they would expire after an hour, so there’d be a clear definition of when your location would be tracked, and when it wouldn’t be.

While everything up until now sounds great, there were serious flaws in the process.

One of them was the fact that locations weren’t really tracked passively, since background refresh didn’t exist at the time, and it was only possible to update locations when the app was in the foreground. This would become an unfortunate restriction that would detract from the app’s overall utility.

In retrospect, one thing I would have done differently would be to design the network stack completely different. Certain aspects of Parse were great, but because it relies on a REST-style API, realtime location updates were slow and innacurate, relying on refreshing after a fixed interval, and only when the app was in the foreground. I suspect a socket-based connection would have resulted in a better user experience.

Almost immediately after launch, iMessage rolled out the ability to send your location, which despite my three simple steps, made it impossible to compete with. While I learned a lot from making Missi0n, resserecting it would require carefully rethinking of goals and objectives.

——

More Info:

MISSI0N Marketing Site

Wobblr was the first big iOS project I worked on back in 2011, when skeumorphic materials and missing vowels/a trailing “r” were essentially required to get onto the App Store.

It wrapped a long lost open-source synthesizer library, which was highly configured to produce a certain sound. By only exposing a few basic parameters to the user, It was effectively impossible to produce something which didn’t sound good.

In mid-2016, Apple yanked it the App Store since it hasn’t been updated since iOS6. I’d love to get it back up, but the source code is long gone on an old computer!