Twiddler – Setup & Configuration

Overview
EDIT: GitHub repo is now up! see https://github.com/ITBatman/Twiddler

“Twiddler” (aptly named by Devin Rader, a Twilio Evangelist) was created as a Fiddler 4.0 extension to simplify the development of Twilio applications.  It allows you to receive a Twilio web hook anywhere there is an internet connection.  This includes public hotspots, or even those created by your mobile phone.  If you have access to the internet, your laptop via Twiddler becomes accessible from Twilio.  I think it’s a fascinating leveraging of technology that still amazes me.

In brief, Twilio applications allows you to use the telephone as easily (or even easier) than you browser.  In fact, with a Twilio, the scope of users that can use your application increases to anyone that knows how to use a phone.   The possibilities are endless, and we are only at the tip of the iceberg!

Twilio works via a webhook, which turns a phone ringing into a web POST/GET to a web server of your choice.  With this web POST/GET, enough information is received so that a  web application can react and build an IVR system (Interactive Voice Recognition).

The catch is that a webhook requires a “publicly accessible web server”.  That means a web server that is typically hosted in the cloud.  However in development, we typically don’t have a public webs server.  Typically we are working with a web server on our workstation, and it’s behind NAT routers and firewalls.  The job of “routing in a request” from the internet is not trivial as it often requires configuration of routers.   This involves Port forwarding, or protocols (like TURN/STUN servers)  to manage the routing.  This all requires a learning curve that typically makes a simple investigation into an overly long process that is simply not that fun, and is typically out of the realm of a hobbyist developer.

I build Twiddler because I am a consultant, and I’m always traveling with my laptop.  If I wanted to do a bit of develop using Twilio, I was unable to do so because I was away from my home resources where I have configured my hardware appropriately.  I wanted something so that I could turn on my laptop, and start working regardless of where I am, and start receiving those webhooks.

Twiddler does that for me via the Azure Service Bus, and much, much more!  This takes no configuration other than simply creating:

1)      An Azure account (free here WindowsAzure.com), and

a.       Establishing a Service Bus endpoint (detailed setup follows). 

2)      A Twilio Account (free Twilio.com),

3)      The fabulous Fiddler 4 (free Fiddler2.com) application, and  

4)      Twiddler  (on https://github.com/ITBatman/Twiddler/)

That may seem to be a long list for a newbie, but it’s taken me 4-6 months of evening and weekends to put Twiddler together to a point that I feel is simple enough to explain in blog post.

Azure Setup

1)      Create a Windows Azure Account

a.       Using a Live ID, create an Azure Account via www.windowsazure.com
Figure1
figure 1

b.      After you have created an account, the intro starts
Figure2
 figure 2

click through each of the tabs for a quick overview of Azure capabilities are.

c.       In the end you should see the main portal page
Figure3
figure 3
Click on the Service Bus, and click on “Create a new Namespace”

 

d.      New Service Bus Namespace
Figure4
figure 4

          i.      NameSpace Name Enter a new namespace name.  This can be anything you want, but it must be unique.  The portal will validate the name, and will indicate if the name you have entered is OK (check mark will appear, see next screen shot) NOTE: Directly under the NameSpace Name, the full URL will be displayed.  All ServiceBus namespaces are suffixed with “.servicebus.windows.net” to create a fully specified HOST.  This will be your “public” endpoint  that Twilio will issue the webhook to.  Twiddler (not Twilio but Twiddler the Fiddler extension) will take all activity directed to this endpoint, and “relay” it to a local workstation URL.

ii.      Region Select the geographical region that you are closest to.

e.      Azure Service  Bus endpoint created!
Figure5
figure 5
and back to the portal
Figure6
figure 6

 

f.        Get a Copy of the Access Key The Access Key is needed later when Twiddler is configured.  The key allows Twiddler to communicate with the ServiceBus and setup relay for you.

          i.      Select the Service Bus NameSpace This will highlight the row and activate the Access Key button.  NOTE: The NameSpace must be specified as “Active” for this to work.  If you just created the NameSpace, this may take 10-30 seconds.

                                                             ii.      Click on the Access key near the bottom of the screen
Figure7
figure 7

                  iii.       Copy the “Default Key” and paste it where it can be recalled later when Twidder is being configured
Figure8
figure 8
Note:  I’ve had to blur out the key, otherwise I may not like my credit card next month!

Your done setting up an Azure ServiceBus! NOTE: Make sure you remember the Servicebus Namespace name, as well as the Default key, as it will be needed later when we configure the Twiddler extension.

Create A Twilio Account

1)      Navigate over to Twilio.com
Figure9
figure 9

Enter your info:
Figure10

figure 10

2)      To verify the Account, Twilio needs to call you.  Simply enter a phone number where you can be reached:
Figure11

figure 11

Click the “Call Me” button

3)      A verification number will display
Figure12

figure 12

And shortly thereafter (like within 30 seconds) the phone number that you entered will ring!  Listen to the instructions, and enter the verification code when prompted

and enter the verification code that was given via the web page:
Figure13

figure 13
At this point in time, the Twilio number that you were given is active, and your almost ready to roll.

If you want to try it out, click on the “Get Started” button, and you will see the following dashboard that will allow you to “test drive” your Twilio number:
Figure14

Figure 14

After you are done, click on the “Go To Your Account” to see the Twilio “Dashboard” for your account.

 

4)      On the Twilio DashBoard, each Twilio account is given an:

a.       Account SID A unique identifier for the an account with Twilio

b.      Authorization Token A unique “password” that, along with the Account SID, allow you to use the Twilio API.
Figure15
figure 15
NOTE: For security reasons, the Auth Token should never be displayed publically.  Click on the Lock icon to see the actual number.  You will need return to this page (DashBoard via the menu bar at top)  as you will need to cut and past the Account SID, and Authorization Token into Twiddler later on.

 

Install Fiddler

NOTE: If you have Fiddler already installed, make sure it’s Fiddler 4.0 (Beta) application.  Twiddler will NOT work with versions previous to version 4.0

1)      Navigate To: http://www.fiddler2.com/fiddler2/version.asp

2)      Download Fiddler 4.0
Figure16

figure 16

3)      Unblock the ZIP file (EDIT: as Eric Law has commented, this is not a required step!)

a.       Right click the filename, and select Properties
Figure17
figure 17

b.      Now double click on the file, and accept all defaults.

c.       Navigate to the Scripts Folder After installing Fiddler, a folder called Scripts will be created in your Documents folder under your profile name (mine is rreukema).  Keep this path Open as you will require to navigate here shortly
Figure18
figure 18
NOTE: Do NOT start Fiddler quite yet… the extension code needs to be copied to this folder before Fiddler is started.


Twiddler

Installing Twiddler

1)      Navigate to https://github.com/ITBatman/Twiddler

2)      Download Twiddler Twiddler is available with complete source code, or only the Binary files necessary to run Twiddler.

a.       Binary Files Only

          i.      Download the file “PutInFiddlerScriptsFolder.zip”as per below:
Figure19

figure 19

          ii.      Save the file to the Fiddler Scripts folder as identified above (and duplicated here)
Figure20

figure 20

                  iii.      Unblock the File

1.       Right Click the file, and select Properties

2.       Click on the Unblock button

    1. UnZip the File
    2. Double click the file, and allow it to expand within the Scripts Folders
      1. Start Fiddler –  The extension will appear as a new tab (see  section on “Configuring Twiddler” for detailed instructions on configuring and how to use Twiddler. (Edit: Tab is now called “Twiddler”)
        Figure21

figure 21
NOTE: If you cannot see this Tab, please see the section on Fiddler trouble shooting later in this document

b.      Complete Source

                    i.      Download all Files

                    ii.      Ensure the following SDK’s are installed:

      1. Azure SDK
      2. Twilio API
      3. Microsoft.ServiceBus
      4. Fiddler.exe Create a reference to Fiddler.exe via the Browse functionality of “Add Reference”
      5. Post Build command The Project has a post build command that copies the Twiddler code from VS directory, to the default Fiddler script folder.  NOTE: If Fiddler is running, the copy command will error out

Configuring Twiddler

Twiddler has three tabs:

  • ·         Service Bus Twidder Parameters (Edit: I updated the name of the tab) Used to authorize Twiddler to set-up a Service Bus relay on your Azure account
  • ·         Twilio Phone Numbers Used to authorize Twiddler to setup Twilio parameters on your Twilio Account.
  • ·         Twilio Debugger Retrieves Twilio Notifications – used for debugging Twilio applications when something goes wrong on the Twilio side of the exchange.

Twiddler was configured this way to provide as much functionality via Twiddler, and minimize the effort to build and debug Twilio applications.  Without Twiddler functionality, I was always jumping between numerous web browsers to debug and develop the application. Twiddler minimizes these interactions so that development can occur between your development environment (Visual Studio), and Twiddler.

 

Service Bus Configuration

The Service Bus tab assists in the creation of the first URL that is used to enter the Twilio Voice URL

Figure22
Figure 22

Field Definitions

As noted during the creation of the Azure Account, please enter the following fields:

1)      Name Space Row

a.       Name Space (first field) Type in the Service Bus name space name as it appears in the portal (without the “.servicebus.windows.net” suffix)

b.      EndPoint Name/Service Path Name (second field) Within in a Service Bus Namespace, you must specify an EndPoint Name, and it can be anything you like (no special characters).  This field is used to create multiple endpoints within a single namespace.  Within Twidder, we only require one.

c.       Controller (third field) This is the name of the MVC controller that the Twilio request will be directed to.   Please enter the name of the controller, with the “controller” suffix removed.  The URL that we are creating for use in Twilio is being created directly below the current line.

d.      Action (fourth field) This is the name of the MVC controller that the Twilio request will be directed to.  This is the Action that will be called when the Twilio phone number is called.l

Directly below these 4 fields, an URL is being created.  This URL is the URL that must be entered into Twilio as the Voice URL of your application (could also be the SMS URL, depending on the type of Application you are creating).  For your convenience, clicking the clipboard button to the right will put this line into the clipboard, make it ready to paste into the Twiddler parameters side of things.

2)      Issuer Name Enter the issuer name as it appears on the credentials popup (default to owner).  When Security becomes a concern, this name should be change when a new issuer name is created with minimal security rights.

3)      Issuer Key As noted during the creation of the Azure account, use the “Access Key” button near the bottom of the Windows Azure portal.

4)      Redirect To: Type in the full root of the MVC or WebAPI host that is listening for requests.  This will typically be Http://localhost/XXXXXX where XXXXX is the name of the web project in your solution. NOTE: if you enter localhost, Twiddler will replace it with “ip4.fiddler”.  This is done to ensure Fiddler directs the request correctly to the localhost and avoids the issue of the loopback adapter.

Command Buttons

  • Start Providing the Issuer Name is correct, as well as the Issuer Key – Twiddler will establish a Service Bus endpoint that is hosted within Azure.  This is a public endpoint that will route all web requests directly to your local workstation.   If the endpoint is established successfully, a message will appear stating “Service Bus listener has started successfully”.  Should an error occur, a message box will appear directly underneath the Start button.
  • End Simply stops the service, and the public endpoint is no longer accessible (and will not continue to accumulate “relay hours”)
  • Save Settings Saves the definition of all fields entered.  When Fiddler/Twiddler is re-started, the fields will be populated from the last time this button was pressed.

Usage

After the “Start” button is pressed, a public endpoint is created using the NameSpace name, as well as the EndPoint name.  The complete “Twilio URL” should be used to configure the “Voice URL” of the Twilio account (see the Twilio Phone Numbers tab for assistance in setting this URL within Twilio).  After the Service Bus Relay has been started, your local workstation is now listening on that endpoint for any requests that may be issued to that endpoint.  Upon receiving that request, Twiddler will redirect the request (doing an URL rewrite), and route it directly to the host that was specified in “Redirect To” ( the NameSpace name, and the EndPoint name are replaced with the “Redirect To” host that was specified).  NOTE: The URL in “Redirect To” need not be sent to a local web service, but could be redirected to ANY web service that is properly configured.  For example, after my solution was deployed to Azure, I choose to have Twilio send the request to the Service Bus endpoint, where I then redirected the request to my Azure account.

Twilio Configuration

Figure23
Figure 23

Field Definitions

  • Account SID As previously noted during the Twilio configuration, copy and paste in the Account SID from the Twilio Account that was created.
  • Auth Token As previously noted during the Twilio configuration, copy and paste the Auth Token from the Twilio Account that was created.  NOTE: Click on the lock Icon first to unlock the contents, which will reveal the code.  Copy this field only after it has been unlocked.
  • Voice URL Twilio will use this URL when someone dials the phone number which is selected on the left.
  • Status Call Back URL An optional URL that Twilio will use when the user hangs up the phone. 
  • Fall Back URL An optional URL that Twilio will use should the Voice URL have a failure of some type.  If not specified, Twilio has a default message that will be relayed.
  • SMS URL An optional URL that Twilio will use should some send a text to the phone number which is selected on the left.
  • SMS Fallback URL Same as the Voice Fallback URL, but only used if the Twilio application has a failure processing the SMS URL.
  • Get Twilio Number URLs During the Twilio configuration, a Twilio number was purchased.  If you click on the button, all phone numbers purchased from Twilio will be displayed in the list box.  The fields on the right will be populated with URLs that were previously saved.  Initially these fields will be blank.
  • Update All Twilio Number URLs This button will update all the Twilio fields for each phone number listed in the list box.
  • Save Settings Saves the Account SID, and Auth Token fields.  When Fiddler/Twiddler is re-started, the fields will be populated from the last time this button was pressed.

Command Buttons

Usage

1)      Click on the “Get Twilio Number URLs” to retrieve all purchased phone numbers on the Twilio account specified (Account SID, Auth Token combination), along with the URLs that may be associated with each phone number.

2)      Select the phone number within the list box.

3)      From the Service Bus parameter tab of Twiddler, copy the “Twilio URL” to the clipboard (click on the clipboard button on the right).

4)      Paste that URL into the Voice URL (or perhaps the SMS URL depending on your application).

5)      Click on the “Update All Twilio Number URLs”.  This will take the current URL information for each phone number, and transfer this information to Twilio.  Should errors occur, an error message will appear directly under the buttons.

(Edit: New Feature – not in screen shot….)

  • Call Via (followed by a text box) This button will SIMULATE a call to your website, and simplifies calling your application.  Enter your phone number (which should be verified by Twilio), and click the button.  NOTE:  The From/To fields of the Twilio post as flipped when using means of calling your site.  As this call is TO the developers phone, and the FROM field is the phone number selected from the list.  This is the reverse of what occurs when the call is made FROM you phone, TO the Twilio application.  This is a work in progress, and I’m hoping to use the computer resources (microphone and speakers) to enable this call, which would elminate this problem.

Twilio Debugger

When an error occurs during a Twilio exchange, Twilio will capture the error on their side, and record it.  This tab allows for the retrieval of those errors without having to bring up the Twilio web site in another browser.

Figure24
Figure 24

Field Definitions

  • Request URL The URL Twilio used in which the error occurred.
  • Request Method Twilio can use either a GET or POST method.  This field indicated which method was used.
  • Message Text The error message as it was retrieved from Twilio.
  • More Info Any type of information that was captured from the Twilio request that may prove helpful in debugging the application
  • # to Get Simply the number of errors to retrieve – errors are retrieved in reverse chronological order.
  • Twilio Debugger Uses the Account SID/Auth Token to retrieve the number of errors specified on the left.  Date and time is listed on the left, and detail information of the error is listed on the right.

Command Buttons

Usage

1)      Account SID/Auth  Token Ensure that these fields have been specified under the Twilio Debugger tab.

2)      Click the “Twilio Debugger” command button to retrieve errors from the Twilio Account.

 

Trouble Shooting

Fiddler

As explained here, Fiddler by default ignores all errors within extensions and silently fails should an error occur.  If you want a little more info from Fiddler, you have to set some preferences in order to display the error messages.  In the “Debugging your Extensions” section, it indicates that you need to set the “fiddler.debug.extensions.showerrors” preference to true, as well as “fiddler.debug.extensions.verbose” preference to true.  In order to do this, type the following commands into the QuickExec box (see here for UI help).

Enter the following commands:

Set Pref fiddler.debug.extensions.showerrors true
Set Pref fiddler.debug.extensions.verbose true

Figure25
Figure 25


Twiddler – Playing Catchup

For some reason I was awake at something like 4:30AM this morning, and could not get back to sleep.  So I picked up my iPad and started reading tweets to put me back to sleep – needless to say that didn’t happen.

@pmhsfelix (Pedor Felix) tweeted out that he had “Just upgraded my Web API HttpServiceBusServer to .Net 4.5, and created a github repository to show his code.  This is exactly what I have accomplished with Twiddler, but there is no way that I could have documented it so eloquently from a technical perspective.  I’m not sure I even understand it from a nut and bolt perspective.  However, I’m sure his technical implementation is much more complete, and more aptly named than my implementation.

He has documented it here in a blog post (http://pfelix.wordpress.com/2012/03/15/asp-net-web-api-creating-an-host-using-azure-service-bus/ ), with the github respository here (https://github.com/pmhsfelix/WebApi.Explorations.ServiceBusRelayHost).  To understand the rest of this post, you will need to at least read the blog post documenting his efforts.

It was a long time ago when I was starting down this road,  and Pedro kindly responded to my tweets/inquires, and I explained what I was trying to do.  At that time he created another version, but still accomplish the same functionality of what he is doing here.  However, at that time I also didn’t really understand the technical implementation.  As I recall, in that first version, he “cloned” the first http request (the one being relayed down from the service bus), and created a second request.   I can’t recall all the details, but in the end it didn’t work for me as expected.  I’m sure that the problem was more on my end than his code!

Since then I was doing a lot of reading, learning more about ServiceBus, WCF, ASP.Net Web API, async programming, and the list goes on.

I then came across this post  by Tony Sneed (http://blog.tonysneed.com/2012/04/24/roll-your-own-rest-ful-wcf-router/), it was an example of how to create a router that circumvents the WCF routing restriction of NOT allowing WCF REST services (see Tony’s blog post for a detailed explaination).  It was the golden ticket, as I was finally able to see in my minds eye a glimpse of understanding.  The key was this interface, and how it allows the handling of a raw Message, without implementing an entire contract of methods:

IRoutingService

(click to see larger, readable image!)

The rest was in his code.

It was from this example that I created Twiddler.   I created Twiddler for the soul purpose of helping me build Twilio solutions.  However, the basic underlying priniciple is to simply expose a public endpoint, and route REST requests to that endpoint down into a workstation, which then issues a reply.

Pedro’s documentation in his post clearly documents the path and complications that need to occur to make this happen.

As I stated in my first post, my implementation is not as eloquent, and it may have bugs, but it’s working for me like a charm.  I’m titled this post catch-up because Pedro is tweeting about his solution, and I would love for him to look it over and tell me what he thinks (as well as @tjaskula, @gblock @darrel_miller who responded to his tweets)

I also think a Fiddler extension that incorporates what Pedro and Tony have done into a Fiddler extension that relys in HTTP or SOAP requests to a local workstation is a perfect packaging of this technology that encapsulates a ton of complexity.  I’m hoping they think so too!

I’m going to release my extension soon, but as I stated earlier it needs a little vetting from Twilio before that can happen.  Hopefully I’ll have it out before the end of the week.

I also have been working on configuration documentation, which I now have to move from a word document with plenty of embedded screen shots, into a web page that can be hosted by this blog.  That shouldn’t take long now!


Working on my first OpenSource project / Twiddler

I know that I have not written much on this blog, but this year is going to be different!  I now have a good job in which I can keep my hours down to less than 40 hours per week!  So instead of relaxing, and attempting to form other hobbies other than this damn computer, what do I choose to do?  Write an open source project!  It is however something that I find fasinating, and since I started on Telephony some 20 years ago, its sure has been fun moving into the world of cloud based telephone systems.  Back in the day you had to buy and get configured a phone number (something like $60 bucks per month), and specialized hardware (like a Dialogic board), in order to set-up a one line IVR system.

Well, things have changed with cloud base providers.  What does this mean? It means you can get into IVR technology for a price point that is obscene!  It’s so low that anybody that is worth their salt as a programmer could easily get into it.  In addition, it may be one phone number, but it does NOT mean one phone line.  In cloud telephony,  you may never get a busy signal!  In addition, with worldwide coverage, your IVR system could easily have an audience that is mindblowing!

My project (actually a couple) uses Twilio.  As I started down the path of my project however, I found that getting started and configured was just a little to hard for my liking.  Twilio leverages a webhook which is basically a callback when an event occurs on a web server.  Twilio’s web hook occurs when somebody dials a phone number (read it here).  After purchasing a phone number from Twilio (they have free accounts), you specify the URL of a webhook which is associated with the phone number.  Thereafter, whenever somebody calls the number, Twilio issues a GET/POST to that URL, and expects an XML package (called TwiML in their terminology) to respond to the phone call.  If you are interested, there are approximately 5 verbs in the XML that give you pretty much endless possiblities on building IVR systems.  See the Twilio site for excellent documentation.

Getting back to webhooks, they have one small fault… a webhook needs a PUBLICALLY available web server.  This might not seem to be a very big ask, as if you want people to find you and your internet resources, you need a web server in the public domain to serve them!  However, when learning a new technology (or during development), the last thing you need is a public endpoint hooked up to a wide open development/training server!  In fact, most of the time the webserver you have is likely to be sitting on a development machine, and there is no way that is going to be hanging out in the public web waiting for a hacker to slice you to threads!

So Twilio issues a webhook, and  your development machine is behind at least a couple of firewalls and likely even “natted” (that will mean something to all my infrastructure friends…).  How can the cloud based Twilio hook up with your lonely development machine?

The hardware path involves configuring routers, DNS servers, and at least one public endpoint that can recieve requests.  I’m not a hardware guy, but I initially went down this route and eventually got this working.  However, the only place it would work was my home office.  If I wanted to develop my little project, I had to be there in order to communciate with Twilio.  However, since I’m a consultant with several clients, employes and a growing family, the amount of quiet time that I get to spend in my home office is minimal.  I wanted to do the development whenever I could find some spare cycles, and do it while I was mobile.

Out of this frustration came Twiddler (named by Devin Radar of Twilio), an extension in Fiddler that allows me to receive a webhook from Twilio easily, regardless of where I am, and how I’m connected to the net.

Look for a several part series that documents how I eventually put this thing together, as well as the availability of the Twiddler as an Open Source project on git (it’s currently being vetted by people far smarter than I).

I also have to say that as a freelance independent developer of software for almost 30 years, I have some thoughts on just letting this software out the door without some kind of invoice attached to it!  It represents many, many months of personnel time learning new technology and techniques.  I take personnel pride in remaining current in technology for nearly 30 years, and that does not come without serious time commitment and effort.  I’m proud of what I’ve done, as many people have tried to solve this juggernut of technology, and I think what I’ve done is eloquent and simple to implement from a functional standpoint (even the implementation may be shit, it still works!).

That said, I could not have done it without the resources available on the net,and the help of those on the net.  I use the net everyday, and it has and will continue to benefit me greatly.  This is my small way of giving something back which I think is worthy of all time when I benefited from others documenting their efforts on the web.  I’m looking forward to where this journey will take me!


IT Batman Arrives!

So why the heck did I name this blog IT Batman!  I named it that after I responded to job that listed several items under qualifications.  These included things like:

  • 12 years .Net Development
  • 7-8 years SQL Server DB Administrator (BI knowledge a plus)
  • Architect
  • SOA skills
  • 9-10 years Technical Lead
  • 4-5 years PM

The list went on and on.  So I called this person and said, that if they found this person that I would pay him/her double the salary that they were asking!  I said if they found anybody with this skill set, they should call him IT BATMAN.

Then it struck me – what a great name!  Totally tongue in check, but I liked it, and now I own it!

I’ll be blogging about mostly technical stuff, especially around SOA, WCF, Silverlight, etc. as distributed application development is what I’m concentrating on lately.  I have an idea for an application that (I think) will need massive scale, and be able to grow quickly.  Hence, the interest.  I’m mostly in the Microsoft stack, so Azure is in my cross hairs.

I’ve tried several times to blog consistently, but I’m a family man, and working in technology.  How much free time do you think I have!


Follow

Get every new post delivered to your Inbox.