Mabel API Is Mostly Built Out Now

I've been hard at work since the last blog post building out the Mabel api. I finished that task yesterday morning. Since then, I've been burning in all the new endpoints and fixing the various bugs that particular process uncovered. 

At this point, the following tasks remain:

• Code the validators for the endpoints that accept post bodies.
• Write all the unit tests for the new/updated code in the application.

After those two tasks are complete, it'll be time to move back to the client and build out all the code it'll need to make requests to Mabel and handles her responses.

Gotta Set Up My Debugger Again

 I can't believe I spent a couple of years not being able to debug code I was running on remote servers. I'm glad this is something I've learned how to do. Those years of debugging via logging were excruciating.

Anyhow, my project in PHPStorm got terminally . . . hosed . . . a few weeks ago and I had to reset it. Something went horribly wrong with the file indexing.

Today I needed to use the debugger again for the first time since that and that meant I had to configure it again.

Luckily, that was easy to do as I found this still germane walk-through from four years ago.

That Feeling When A Bug Makes You Happy

It's not very often that you come across a bug in your code where, after tracking down the source of the problem, you're pleased that the bug exists.

According to the documentation on redis.io, the hgetall command returns all the fields and values of a hash at the given key. However, the command result is a list where each field is followed by the corresponding value. 

I assumed that calling this method in predis would return a result formatted the same way and I baked this assumption into my code.

When I hit an endpoint that executed this code for the first time the api returned an error response and I could see an exception in my log file. Looking more closely at the stack trace, I could see that Mabel was attempting to access numeric array indexes that didn't exist.

I threw in some logging and saw that, contrary to my earlier assumption, predis was automatically translating the result of the hgetall command into an associative array.

This is great news. I didn't mind working with the result format described in the redis.io documentation, but I knew that if I were dealing with larger hashes I might start to run into problems. Working with associative arrays is much easier.

So yeah, sometimes a bug in your code can be a good thing.

A Use Case For Middleware

With the exception of /anonymous/news, every valid request to Mabel has to go through the "check if the authorization header is in the request, get the header value, is the value an empty string, yada yada yada" rigmarole.

Initially, I put this authorization functionality into the beginning of the action() methods of the controllers I was building out. Eventually, I noticed commonalities and moved that code up to the base class that all controllers inherit.

It became clear though that I was going to end up in a situation where controllers of the same authorization case would contain identical code at the start of their action methods. However, I would not be able to move that code up to the base class without implementing some really 'clever' code. And as I always say, "there's good clever and there's bad clever" and this would clearly have been a case of the bad flavor.

As it so happens, this is the scenario where middleware really shines. Middleware is simply the idea that I can tell Slim (or other similar micro-frameworks) that I want code to execute before or after the action in a controller. In some frameworks you explicitly state "I want this code to execute before" or  "I want this code to execute after". In Slim 4 you sort of just wrap your middleware around the action and handle the 'before or after' part in the middleware itself.

Ultimately, I was able to move all of the repetitive code out of the controllers entirely, leaving behind just the code that pertains specifically to that controller's action.

So yeah. Yay middleware.

 

Refactor Everything. Then Refactor Everything Again.

This post has been very difficult to write. In some sense, the process of writing it mirrored what the next step of building out Mabel was really like. I knew that the end result of the post was going to be a description of the final User model that I developed and how it worked. Similarly, I knew that the end result of "the next step in building out Mabel" would result in a concise, unified and rational framework for managing persisted user data. What I didn't know in either case was how to get from start to finish.

With respect to developing the User model, what really helped was being able to generate the following set of requirements that the above mentioned framework must implement:

1. If I load user data via Redis\User I must do so with the value of the authorization header.
2. If I load user data via MySQL\User, I want to save the data via Redis\User so it can be used later as the fast source.
3. However, in some cases I need to perform validation on the data loaded via MySQL\User before I save it via Redis\User.
4. In some cases I wanted to load data via MySQL\User with the User Name field. 
5. In some cases I want to load data via MySQL\User with the User Id field.
6. When I load data via MySQL\User, I want to save the User Id via Redis\Authorization as cases exist where there would be no other way to link the an authorized header value to the actual user account.
7. When I save data via MySQL\User and am creating a new record, I need to confirm that the User Name, Screen Name and Email Address do not exist for any record currently in the database.
8. When I save data via MySQL\User and I am updating a new record, I need to confirm that the User Name, Screen Name and Email Address do not exist for any record currently in the database EXCEPT for the record of the current user.
9. When I save data via MySQL\User, I want to save the data via Redis\User so it can be used later as the fast source.
10. When I save data via MySQL\User, I want to save the User Id via Redis\Authorization as cases exist where there would be no other way to link an authorized header value to the actual user account.

Once I fully understood what the requirements were, coming up with the implementation was pretty easy. Generally speaking, the code is structured as follows:

• There are three user models
• Models\User
• Models\Redis\User
• Models\Database\User
• Models\User requires the other two as constructor parameters.
• All three models have identical public getters/setters for user data.
• Models\User has three methods that can be used to load user data:
• loadFromUserKey
• loadFromUserName
• loadFromUserId
• Models\User has two methods that can be used to save user data:
• saveToRedis
• saveToDatabase
• It's the controller's responsibility to manage which load or save methods to call and in which order. 
• It's the responsibility of Models\User to manage the transfer of data between itself and Redis\User or Database\User as appropriate.
• It's the responsibility of Database\User to manage the special validation that occurs when saving user data to the database.
• It's the responsibility of Database\User and Redis\User to manage the actual communication with their respective services.

It came together pretty neatly in the end and I feel pretty good about it. There's very little duplication of code. I expect it'll be relatively straightforward to unit test and it's not terribly confusing code to read through.

Definitely Going To Implement A Fast/Slow Persistence Model

One of the things that I want to implement for Mabel is fast/slow data persistence. 

It works as follows:

• When data needs to be loaded, the first thing that's checked is a 'fast' persistent store like Redis. If the desired data isn't located in the fast store a slow store, like a MySQL database, is then checked.
• When data is saved, the data is first saved to the slow store. Then the old data in the fast store is replaced with the newer.

Why would I want to implement this? Well, slow is . . . slow. In the case of a MySQL database, you really don't want every single SELECT statement in production to execute against the database, especially since a large number of those statements return identical result sets. Might as well cache those identical result sets somewhere and take the load off the system.

If I were going to do a full implementation of a fast/slow system I would also put the slow system updates behind some sort of a queuing tech like RabbitMQ or something bespoke built on top of Redis. That's really only necessary when you're very very concerned about your slow store being overwhelmed by the number of writes it's being asked to do or the update to the slow store needs to trigger other back-end processes that you don't want handled synchronously (like sending emails).

At this point in time, that's not necessary so I'm not going to build it.

But yah never know. Maybe some day . . .

How Exactly Is Authorization Going To Work?

After reorganizing Mabel to reflect the decision to move my dependency declarations out of docroot/index.php and to switch to a 'one endpoint one class' approach, it was time to dig back in and go to an even greater level of detail.

Authorization seemed like a good place to start but I also wanted to flesh out the data model for user information, so I began with POST /account/create.

The business logic for this controller should be something like:

1. Confirm that the request has the authorization header and throw an exception, log it and return a 400 Computer Says No response if it doesn't.
2. Get the value of the authorization header.
3. If the value is an empty string, generate a new value and save it to persistent storage in an unauthorized state.
4. If the value is not an empty string, attempt to load it from persistent storage.
5. If the value cannot be loaded from persistent storage, generate a new value and save it to persistent storage in an unauthorized state.
6. If the value exists in persistent storage in an authorized state return a 403 Computer Says Forbidden response. You should not be attempting to create a new user account if you're already authorized.
7. If the value exists in persistent storage in an unauthorized state we're golden. Continue on.
8. Get the post data from the body of the request and translate it into something the code can work with.
9. Validate the translated data for the existence of all required fields, that those fields don't contain invalid data and that they're neither too long or too short.
10. If the translated data failed validation, return a 406 Computer Says Not Acceptable response.
11. Confirm that no other already persisted user has the same User Name, Screen Name or Email Address included in the translated data.
12. If one or more of these three fields are already in use, return a 406 Computer Says Not Acceptable response.
13. Save the translated data to persistent storage, thus creating the new user account.
14. Update the authorization header value saved in persistent storage to an authorized state.
15. Return a 200 OK response.

So yeah, that's basically what's happening behind the scenes for that particular endpoint. The other endpoints are largely variations on the same theme.

Ohhhh. So That's What Autowiring Is.

 I finally had that 'eureka' moment with respect to the Slim 4, PHP-DI and autowiring.

After roughing out the controllers for Mabel I realized I had a small problem in that the api didn't return a 404 response when a uri was sent that didn't exist.

Like, if the client asked for /foo/bar the api would return a 200 OK response. That's all sorts of wrong.

So I started to dig into the documentation a bit trying to figure out how to implement the correct behavior. As part of my research into that, I ran across the following blog post on error handling in Slim 4. 

That post contained a link to the slim-skeleton github repository, which I found pretty intriguing. In particular I really liked how it manages dependencies and routes. Up until now I'd put all the code loading up the container straight into docroot/index.php, but I've never liked that. Seeing an alternative approach that I did like, I just copied it.

I also really liked how each endpoint corresponded to a separate class with a single action() method.  I feel that writing unit tests will be more straightforward for code structured this way and I also like how the file structure of the project literally tells you where to find each method for each endpoint. So I copied that too.

I still wasn't fully understanding how the implementation worked though. Looking at the parent UserAction class wasn't much help. Looking at the parent Action class was, however, something of an eye opener. At some point Slim 4 must be executing the ListUsersAction class (or any other class that extends Action) as if it were a function. That was triggering the __invoke magic method, and when that happened the Request and Response objects and the request arguments were being passed in.

After I figured that out I assumed I'd go to app/routes.php and see where all the dependencies were being instantiated and passed to the constructors of the various action classes. Only that's not what was happening at all. Nothing was happening. The route was being defined and linked to a specific class that wasn't even instantiated. 

It was as if . . . ohhhhhhhhhhhhhh.

Slim 4 is using reflection to determine whether a defined route is being linked to a closure, a class or the method of a class. It's then executing that closure, class or method as a callable and passing along the same three parameters every single time. 

And if Slim 4 is doing that, it must also be using reflection to inspect the constructor of each class it's asked to instantiate and using the type hinting of the constructor parameters to search the container object for the corresponding closure that provides that type hinted class in its instantiated form.

So that's what auto-wiring is.

Let's Add Some Controllers To Mabel

And now it's time to build out the actual api for Mabel.

Looking at the documentation for Slim 4, I'm going to allow Slim 4 to instantiate my controller classes for me. This allows me to write the controllers in a manner that's similar to what I've done in the past. 

Namely, I'm going to create controller classes that contain methods that are related. In other words, I'm going to create three controllers: Authorization.php, Accounts.php and Anonymous.php.

Then I'm going to add routing to docroot/index.php that explicitly links each URI path to a specific class and method that should be invoked.

Somehow, Slim 4 magically passes the Request and Response objects and the request arguments to each method in each controller. 

I'm also magically able to have the Container object passed into the constructors of the controllers and that's how I'll get the dependencies I'll be using in the controller. I don't really understand yet how any of this magic happens (which I'm never comfortable with) but I suppose that's not important right now.

There's something called auto-wiring that's being utilized, or at least it's an option, because I'm using php-di as my container. I don't understand what that is or why you'd use it. The php-di documentation really talks it up, but I prefer not to use something I don't yet understand.

A Couple Of Things I Learned About Unity

I learned a couple of simple things about Unity while putting together the basic pre-game UI.

1. Unity is very easy to use

It's VERY easy to construct simple game elements in Unity. Adding the visual game elements, writing code behind those elements and wiring the two up is very straightforward. I had expected that prototyping the pre-game UI would take a whole week. That I was able to do it in less than two was quite a shock. I suppose it helped that I've already worked through two tutorials that both took something like a full day of work to get through. I should point out that I haven't actually written the code that sends requests to the API and handles the responses from it. I expect that will take quite a bit of time.

2. Panel UI elements are your friend

When you need to show or hide groups of UI elements the recommended (according to a forum post I agreed with) way to do that is by containing elements in panels and showing or hiding the panels. I'd already sort of started putting the UI together like that when I came across the guidance. Grouping elements by panels also makes it a lot easier to structure the layout and ensure that it generally continues to work if the user operates at a different resolution than you do.

3. Unity is magical. Use that magic.

I ran into some problems when I started to actually write the code that made specific elements show or hide (you're actually calling GameObject.IsActive(bool newState) method to do this).  There are methods in Unity for finding the child objects of UI elements by name but I wasn't looking forward to writing a bunch of "traverse tree structure for child element with name" code. Luckily, I came across a related question in the forums that finally made an aspect of Unity click for me in a way that it hadn't before.

In Unity, you can write a class in C# and then add that class as a component to a game element in the UI. If you add public properties to that class in code, those properties magically appear in the game design UI as fields of the component. You can then drag other game elements of the UI to those fields. References to the corresponding game objects then become available to the C# code. 

It's freaking magical.

As is Google searching.

Towards A More Advanced User Interface

Setting up the UI elements in Unity was pretty straightforward once I knew the structure of the API that the client was going to be communicating with.

Here it is in action. I should point out that the client isn't actually wired up to Mabel yet as the endpoints don't currently exist.

It's not pretty. I'm not an artist and I plan to generally ignore the aesthetic elements unless I'm inspired or have no other choice.

Defining The API Endpoints

After figuring out the basics of how to build the pre-game portion of the Unity application, I spent some time in front of the whiteboard (in reality it's a cork board I pin notecards into) figuring out what Mabel's API will look like.

These are, from my experience, the endpoints you need for a minimalist user account management api:

POST    /authorization/login        unauthorized only
POST    /authorization/logout       authorized only
GET     /accounts/view              authorized only
POST    /accounts/create            unauthorized only
POST    /accounts/update            authorized only
POST    /accounts/passwordupdate    authorized only
POST    /accounts/passwordreset     unauthorized only
GET     /anonymous/news             no authorization functionality

For all endpoints except /anonymous/news, the API expects the presence of a special header to be present in that request. 

When the request enters the API, that header value will be used to look up whether or not that value is authorized. 

If the authorization state does not match what's in the third column, the request will be rejected and an error response returned.

If no header value is present or the value doesn't correspond to anything looked up by Mabel, a new header value will be returned in the response, even if it's an error.

The /anonymous/news endpoint isn't part of the account management api (which is why it won't implement the authorization system) but I thought it would be cool to have the ability to display 'content' on the pre-game portion of the app. I'll probably try to figure out how to get it to return a formatted version of Mabel's git log.

Well That Was Easier Than I Expected

So yeah, I didn't expect to get this first iteration done so quickly. 

At a VERY rudimentary level I've managed to create a client application in Unity. This client application has a some input fields and a button. When you click the button, the values are read from the input fields and then written to the console. After that, a request is sent to Mabel. Once the response from Mabel is returned and if it was successful, the response body is printed to the console.

Initially I found a short youtube video that was very helpful in just getting me pointed in the right direction in terms of how to create the UI.

The Unity documentation was pretty well structured but I didn't really have to rely on that very much yet.

When it came to writing the C# that handled the remote api call, I was able to find an online reference that allowed me to put together some very rudimentary code that worked almost immediately.

I'm definitely going to need to get my hands on modern references for C# and related best practices. It's been a while since I've written anything in it.

In particular, it looks like the C# libraries for handling remote api requests are very heavily tilted towards being asynchronous. This is fine (and good), but not something I'm extremely familiar with. I had to deal with this years ago when I wrote a basic nodejs server application, so I'm comfortable with the concept. I just don't know how the C# syntax for this fits together and what the best practices are. Something new to learn, I suppose.

Next step is to flesh out in more detail what the client application needs to do with respect to starting the game for the first time (new account creation, etc.) and what that implies with respect to the C# code that supports it.

Connecting To Mysql 8

I actually instanced the MySQL 8 database I'm planning to use via RDS back in early November. The security groups for the DB are setup such that it cannot be accessed by anything outside of my VPC. Both the http server and the bastion host can connect to it, but any manual SQL should only be executed from the bastion host.

To get this all up and running there were a couple of things I needed to do.

1. Update the service ini file so that it contains the correct credentials for connecting to the database.
2. Install the myself command line shell on the bastion host so that I can manually execute SQL.
3. Execute said SQL in order to create the necessary schema, user and table.

The first step was easy. I did run into a problem when I attempted to setup an alias for the DB host via the /etc/hosts file. For some reason that didn't work. Some quick googling suggested I check my file permissions, but as expected, the file is globally readable. Oh well. I just ditched that idea and moved on.

The second step was . . . more complicated. Amazon linux extras doesn't come with any mysql topics. So, I had to do some research to find a yum repository that contained what I needed and then figure out (as a yum n00b) how to actually make it so that the repository in question can be accessed.

First I found some AWS documentation that led me to believe that Amazon Linux 2 is roughly compatible with RHEL 7.

Then I found some mysql documentation that seemed to imply that if I downloaded and installed the correct rpm package, I'd be able to access a yum repository for MySQL software that's compatible with RHEL 7.

Once I did those two things, in theory I'd be able to find and install a version of the mysql client for MySQL 8 that would work on my bastion host.

And that turned out to be the case.

After that, I ran the SQL to setup the schema, user and table and VOILA my new API is fully functional (such as it is).

It doesn't do much yet, but I'll be able to use it to develop the foundation of the client application.