A Long Post for a Simple Rule

The other day, my buddy Ryan asked me a question about a program he was working on:

Ryan: so say I have three lists of (different) objects in a helper class to deserialize JSON to
Ryan: then these objects get saved to a database
Ryan: in the current version of our platform, all three of these lists will be populated, but in the previous version only two lists get populated
Ryan: would it be better to create a new helper object just for the backwards-compatible API call? or use a variety of if statements to check whether that one list is null or not and update the DB accordingly?

 As I mentioned in my post on the XY problem, I needed to get more into the details about the problem:

Ed: umm
Ed: depends?
Ed: you're adding a new list to this object?
Ed: or was it always there, but is sometimes filled?
Ryan: basically JSON data from a HTTP request is automatically deserialized into an object, and depending on API version a client will either send 2 or 3 arrays of objects
Ryan: would you create two different objects to deserialize to or use one object and just check whether one of the lists has been populated

At this point, I was pretty sure what it was he was trying to do. He was rewriting a service with a new API, but it had to be backwards compatible. But we were still talking about a problem one level removed from the important issue at play: changing the API.

Ed: is the api changing? is the client changing?
Ryan: the API is changing, but the client may not change
Ed: So the question you have is, On the backend of the API, do you create separate classes to represent data for APIv1 and APIv2?
Ryan: right
Ed: what does APIv1 send
Ed: right now
Ed: 2 arrays or 3
Ryan: v1 sends 2, v2 sends 3

This is a sneaky issue. One might think that changing an API to send additional data back is not really a breaking change. I mean, if they're still getting X and Y, who cares if they get Z as well?

You should care! You can't predict how someone else will use your API. Maybe they'll write some terrible code that would throw exceptions if there are three arrays returned. Even if the consumers of your API are your coworkers, you have to expect that your consumers will rely on every single thing your API does.

And so, I began to dissuade Ryan of his folly:

Ed: so if you use the same object, you'd change APIv1?
Ryan: no, v2 only extends on v1, doesn't change it
Ed: but v1 sends 2 arrays now. if you make it send 3 arrays, you are changing v1
Ryan: I'm deserializing to the same object via two different controllers in two different namespaces
Ed: doesn't matter. Don't change v1
Ed: and changing the return value, even if it means adding an always-empty array, is changing the API

Like I said, this one's sneaky. We as programmers tend to think about how we would use our own code. But an API, even if only available to a limited set of people, is a public interface, and people will do dumb things with it.

Conclusion:

Ed: a good rule of thumb is Never change the API
Ryan: yeah I guess I would be changing it, wouldn't I
Ryan: damn
Ed: yes
Ed: that concludes today's lesson

The XY Problem and the Five Whys

Have you ever heard of the XY problem?

What is it?
The XY problem is asking about your attempted solution rather than your actual problem.
That is, you are trying to solve problem X, and you think solution Y would work, but instead of asking about X when you run into trouble, you ask about Y. 

The Problem
This can lead to frustration by people who are trying to help you solve the problem because by the time you ask about it, the solution that you need help with might not have any obvious connections to the problem that you are trying to solve.

It comes up a lot, and it's something you should try to recognize both in yourself and in others. I recently encountered it with a coworker of mine:

Coworker: Have you used the HtmlAgilityPack?
Ed: a little
Coworker: I can't get it to work.
Ed: what part of it
Coworker: Sitecore apparently has it included?
Ed: it does
Coworker: But I can't reference it.

I was perplexed at this point. HTMLAgilityPack is an assembly included by default in Sitecore installations. In fact, Sitecore itself relies upon it and will not work without it. So I probed a little deeper:

Ed: the project should already include a reference to it
Ed: ok
Ed: so, when you reference it with the using statement, its just not available?
Coworker: Oh.
Coworker: It looks like the sample code I got needs a later version?

Now we're getting somewhere. The problem wasn't that my coworker couldn't get the HTMLAgilityPack to work. Instead it was that he had the wrong version of the assembly. We then thought about upgrading the version of the assembly, but before we did, something struck my mind:

Ed: do you need the latest HTML agility pack? 
Coworker: Probably.
Coworker: At least, the one included doesn't have methods I need.
Ed: what are you trying to do
Coworker: So, I'm trying to take a substring of content, to display in a "Featured Pages" section.
Coworker: But if there are any tags that open in the substring, but close after, the formatting breaks.

Now we're really getting somewhere! We now were both on the same page about what he was trying to do (already a far cry from "I can't get the HTMLAgilityPack to work"). From here it was just a hop skip and jump away from monkey-patching in the method he needed from the newer assembly (for those with a desperate need of closure, the method was "Descendants()"). 

This whole exchange reminded me of the Five Whys:

To reach this sweet spot, we borrowed an idea from Sakichi Toyoda, the founder of Toyota. He calls it Five Whys. When something goes wrong, you ask why, again and again, until you ferret out the root cause. Then you fix the root cause, not the symptoms.

This is basically what the XY problem boils down to, a lack of finding the root cause/problem. However, simply asking "Why" won't really get you to the proper solution in the XY problem case. Employing pure "Five Whys" in this situation would not have gone so well:

• I can't get the HTMLAgilityPack to work. 
• Why? It won't compile.
• Why? The compiler says the methods I need from sample code I found aren't there.
• Why? The sample code was using a different version of the HTMLAgilityPack. 
• Why? I don't know, man, ask the author of the sample code!
• Why? I can't read the man's mind! I don't even know him!

I find it helps to modify the question from "Why?" to "Why is that important?" or "How so?" or "What do you really mean?" when just asking why wouldn't work. . 

To summarize my interaction with my coworker in my modified "Five Whys" method (call it the XY5Y method) would look like this:

• I can't get the HTMLAgilityPack to work. 
• Why? It won't compile.
• Why? The compiler says the methods I need from sample code I found aren't there.
• Why? The sample code was using a different version of the HTMLAgilityPack. 
• How do we fix this? Let's upgrade the version of the HTMLAgilityPack.
• How would that solve the problem? It would give me the methods I need to solve my problem.
• Are the methods all you need? Well, yes...
• Is there another way to get the methods? We could disassemble the newer version of the HTMLAgilityPack and monkey-patch in the methods we need
• Profit!

If you find yourself working with a new programming language or framework (or really a new anything), make sure to question yourself (or your comrade) to find what the real problem is. 

One of the Little Things That Makes Me Rage

I've been trying to learn my hotkeys for Visual Studio so I can operate much faster. The most common one I use these days is attaching the debugger to IIS. For years, in Visual Studio 2008, I would type "alt, d, p, w, enter". That would open the debug menu, select "Attach to Process...", select the first item that starts with "w" (almost always w3wp.exe, the IIS process), and attach to it.

Then came along Visual Studio 2010. I don't know what changed or why is changed, but now, I have to hit an additional "enter" in the middle: "alt, d, p, enter, w, enter". When using both VS2008 and VS2010 at the same time, this can get really freaking annoying.

resharper - surround with

I recently discovered a really useful Resharper command for Visual Studio: "Surround With". Simply hit "Ctrl+Alt+j" and you get a nifty menu with a list of common things your would surround code with. This feature makes programming simple: write the code, then add the hardening later. Booya!

Visual Overload

I reached my record of running five concurrent instances of Visual Studio (four 2010, one 2008) recently. Seems like there should/could be a better way to have multiple solutions open at once, especially since each VS took about 500MB of memory each (with 8GB memory total, this wasn't really a problem, but it could be if I also had other high-memory programs open). Maybe one VS can have multiple solution-spaces open, like tabs, but at a higher level than the tabs for open files? Not sure, but it seems like a better answer is waiting for someone to discover it.

Building Integration Tests With Workflow Components

A lot of blood, sweat, and tears have been shed over the use and design of unit tests, but what about the red-headed stepchild of automated testing, integration tests? They have fallen out of vogue lately in popular discussion, but they can be just as useful as unit tests. Just search for unit test best practices and integration test best practices; the unit test search has almost twice the results.

One of the biggest stumbling blocks I have found with integration tests is internal dependencies (external ones, like databases and webservices, are also a pain, but you’re on your own for them). If you have to test an “Approve” action, you probably first need to “Create” the object before you can approve it. You’ve written your “Create” integration tests, so do you make your “Approve” tests inherit from the “Create” tests? That seems kind of ugly. But you don’t want to have duplicate code, even if it’s just in the tests.

A better solution is to split your integration tests into “workflow” classes. Each workflow class represents an action you want to test, “Create” in this case.

public class CreateResults   
{
  //data to return    
}    

public class CreateWorkflow    
{    
  public CreateResults DoAction()    
  {    
    //Do the action, return the results    
  }    
}

You can use this code for your “Create” tests, but even cooler, your “Approve” workflow can consume the “CreateResults” object:

public class ApproveResults 
{
  //data to return  
}  

public class ApproveWorkflow  
{  
  public ApproveResults DoAction(CreateResults results)  
  {  
    //Do the action, using the data from the create test, return the results  
  }  
}

How cool is that?

Well, what if you want to have a workflow with various options? We can do that:

public class CreateOptions   
{    
  //inputs to the workflow
  
public class CreateWorkflow   
{    
  public CreateResults DoAction(CreateOptions options)    
  {    
    //Do the action using the provided options, return the results    
  }    
}

You can experiment with using the “Options” and “Results” objects in different ways. YMMV. This may not solve all the problems with integration tests (and it’s not revolutionary, and someone else probably already did it), but it can make for some pretty modular tests and make your life easier.

Look At the Time Stamp

Seriously, just look at when I posted this. Not doctored at all.

I just finished a daily merge for work. Yes, I took time off when I got home to eat dinner and watch a little TV. I still spent a whole day + some on a single daily merge. 

Sometimes, I just hate source control.

Why I Don't Like SOLID Principles

SOLID princples:

• Single responsibility principle
• Open-closed principle
• Liskov substitution principle
• Interface segregation principle
• Dependency inversion principle

This topic has been discussed to death in the tech circles, thanks to the whole tiff started by Jeff Atwood and Joel Spolsky. I'm just throwing my piece in the ring:


SOLID is good.
SOLID is not gospel.

The first rule of programming should be "use your brain." The first answer should always be "it depends." SOLID is not a silver bullet (because, as we know, there is no such bullet). There are going to be times where you need to write some code that is not pretty. Sometimes we acrue technical debt for valid reasons. Usually the reason is we need to implement some feature or bug fix fast, possibly because we're losing customers.

It really comes down to what Jeff said: "Quality really doesn't matter that much, in the big scheme of things." What matters is what we deliver to the customer. Of course, good code quality helps you deliver more and better apps to the customer. But the customer does not care that you followed the SOLID principles to the letter as long as the product is right.

Joel Vs. Kent Smackdown!

I've been listening to the Stackoverflow podcast for a while now and reading JoelOnSoftware for much longer than that. Joel recently started somewhat of a fight by laying the smackdown on SOLID design principles and 100% unit test code coverage.

Kent Beck was not fond of Joel's (self-admitted) generalization of SOLID principles. You can't really blame Kent, since Joel basically just dismissed Kent's whole deal. 

Joel said that the client-specific interfaces principle was probable designed and practiced by people who don't write much code. Kent did not go into details about why he disagreed, simply lamenting that Joel doesn't understand his own responsibility as a popular personality. 

Now, we could debate the merits of this until our ears turn blue*, but that's not what I have to add here. For people who are supposed to be leaders in their fields, they're not really thinking about things. Joel's most obviously talking about FogCreek-style software; shrinkwrap software that other software does not interface with. You know, where there is no client to have an interface with**? 

All I'm saying is that we need to use our brains before we talk. That's all.

*I happen to come down on the side of Joel on this one, except for public APIs (which Joel acknowledges as an exception as well. 

**Except the human client, but that's a whole other can of worms.

The Balance Between Safety and Speed; or, Does the Turtle Really Beat the Rabbit?

Eric and I argue about programming all the time and about every little thing, but we're really just arguing about one thing. I value safety, and he values speed. Let me elaborate.

The safety I'm talking about is making sure my code works right. I'm for strong typing because it can help find errors at compile time. I believe that databases have to enforce their constraints as best they can even if you're the only client. I prefer centralized version control (CVCS) over distributed version control (DVCS) because the restrictions CVCS enforces promote healthy branching and code. I believe in continuous integration, including full unit and integration tests.

The speed Eric talks about is being able to write "5.times.per.day" in Ruby and have it just work the way you expect. Eric loves git because he's able to get input seamlessly from developers around the world and easily integrate new code. And he loves easily discoverable features of his languages of choice.

Most of the time, the programming context determines the value of the two assets. Building a mission-critical app that handles financial data? Safety rules the day. Building a site to determine if it is Christmas or not? Speed please.

Programmer personality also is a significant factor. I know I make mistakes, so I prefer to use tools that prevent/detect said mistakes. I also know that others might use my code at some point, and as anyone who has ever worked with someone else knows, other people are dumb.

The reason I just started writing about this was because I had to change a "Thread.Sleep" call in one of my integration tests from 2 seconds to 5 seconds. To make this change and formally deploy the code to my project's development environment, I had to run the code through the entire 12 minute build process. And that's AFTER running the unit tests locally (5 more minutes). It would've been so nice to circumvent the whole process. And I wanted so much to do just that! But I held fast to my rules, and now I know the published package built from that build is ready to be released to QA.

Domain Squatters, Be Alert

I'm probably going to register edschwehm.com in November at some point... Can't justify it now, but I want to get my home server set up first and I need time to do that. Said server is probably not going to be slackware anymore, thanks to Falcon being all .NET. Good thing Movable Type works on Windows too.

Walk the Walk!

So I talk about development here. F#, nullable types, Walrus, Falcon... but what have I done lately? Nothing!

I have so much crap going on in the fall that I just get drained of energy. Pep band on Tuesdays and Thursdays, plus the games on Saturdays. Winterizing the home. Softball games.

Well, it all ends in November. Pep band and softball will be done, plus the house should be mostly winterized. If I don't start being productive by my birthday, I'm in trouble.

F#: For Me?

So I realized that Walrus could possibly use F# for the rules engine for recommendations. How cool would that be? I could have a second project in F# that takes the draft object (containing the team rosters, draft order, league settings, and free agents) and spits out the recommendation list. I think that's what I'll do.

Don't Visit This Site!

I once interviewed at TripAdvisor.com. I wasn't especially impressed by the interview, but I chalked that up to my own inexperience. I didn't get the job because the CEO misunderstood a question.

I asked, "TripAdvisor.com looks pretty complete to me. What features do you plan to add in the next 1-5 years?"

He heard, "TripAdvisor.com looks pretty complete to me. Why bother doing anything else?"

Recently, a friend of my interviewed there and was rebuffed as well. Though he managed to not offend the CEO, he was tripped up by several weak interview traps (e.g. asking him to write code to determine if a number is a triangular number, but not telling him what a triangular number is). He didn't get the job either.

Upon hearing about adventures in interviewing at this company, I decided to take a look at their website and see how it has changed.

I can't say I'd ever use the site. Even with AdBlock enabled, the site is still cluttered with advertisements. I tried searching for Tahiti; the site was just so ADD I couldn't parse any of it. It's so Web 1.0* it hurts my eyes.

What bugs me the most is that they actually seem to be doing very well! Look at these Alexa rankings:

It's beating the sites where you can actually book travel (ok I get it, travel research and travel booking are not the same category of webpage)! And in case you're wondering, that's good for a 414 ranking! I mean, this is just ridiculous for a terrible website! I'm just blown away by how people can actually use that site.

* One of the questions they asked me in the interview was How would I go about adding a table to a page. I described the table syntax, and the dev manager was all "No no I meant what would you do?" I tentatively asked "Well, if it was just HTML changes, I'd do the changes on a dev page, then copy that page to production." He turned to the computer in the office and asked me to demonstrate on the Tripsadvisor.com homepage. Thinking it was a trap of some sort, I was careful to explain every step as I saved a copy of the live page, made the change, saved, then opened in IE. He nodded then explained that's how they made many changes. This was their dev process? Yuck!

New Project, Codenamed Falcon

I've started working on a new project. It's codenamed "Falcon." Why? Because that was the first animal name I thought of when I decided to name it after an animal. Now I've got Walrus and Falcon.

I have to keep mum on Falcon though, because I actually think it could be really cool.

In The Spirit Of Sharing Information: SQLBulkCopy Class in .NET

I had an annoying problem at work the other day; how to copy a subset of data from one SQL Server database to another. This set included data from six tables. I looked at my options:

• bcp (bulk copy) command-line utility
• Dynamic SQL (select "insert into items (" + items.* + ")")
• write a c# app to load the data into a dataset, then insert it into the new DB.
• Use the Import/Export Wizard.
  

The first three didn't look particularly appealing, and the wizard only exports queries to data files; I'd have to do a lot of work to get those files to work.

Then I discovered the SqlBulkCopy class. In the .NET Framework 2.o and beyond, the SqlBulkCopy class gives bcp functionality in the managed code world. It made my life much easier.

God I Am Such a Nerd

I recently bought an MP3 player almost exclusively for the Stackoverflow podcast. I don't know why I wanted it so badly, but that's another story. I started listening to just SO and NPR's "Wait Wait Don't Tell Me", but I added like 8 more to my sync list (thanks Mark!). I listened to DotNet Rocks this morning. They were talking about Parallel Extensions to the .NET framework and WPF, and all I wanted to do was run home and start rewriting Walrus to use all these fun new tools. I'm such a nerd.

Stackoverflow is Awesome

I'm up to 245 rep with 8 bronze badges and 1 silver (beta). Sweet.

Classic Pat

From: Ed
To: Pat
Subject: Labels folder

Hey Pat,

Is there a reason you moved the instructions files from the "instructions" folder to a folder called "labels"? Just curious. Thanks!

-Ed

---

From: Pat
To: Ed; Coworker1, Coworker2
Cc: ProjectLeader
Subject: RE: Labels folder

I moved them into their own folder because they seemed like files that:

1. Didn't need to be in the same folder as the other files

2. wouldn't need to be changed much

3. we didn't need to look at them everyday in the file list.

---

From: Ed
To: Pat; Coworker1; Coworker2
Cc: Project Leader
Subject: RE: Labels folder

My only question is why was it called labels; the idea is definitely good, which is why I put the instructions in a folder called "instructions".

---

From: Pat
To: Ed; Coworker1, Coworker2
Cc: ProjectLeader
Subject: RE: Labels folder

I couldn't think of a better name and they seemed like they were 'labels' at the top of all our forms...

What's the Big Idea, Fox?

I've been thinking about databases a lot lately. It's been sort of related to work and sort of general, but whatever the cause, databases have been in my mind lately. Then, as my mind often does, I started a thought tangent, where my thoughts quickly bounce from topic to topic, and I ended up on Star Fox.

And then from Star Fox, I went to Falco. And from Falco, I remembered this quote:

"You worry about your own hide!"

And it was that moment that I decided my next database would be named "Falco".

I've mentioned before about how I think that databases should be responsible for their own data integrity and not rely on application code (or developers) to maintain it. Somehow, Falco's philosophy just fits that perfectly.