NetworkGremlins

It’s been a bit since the last post but, again, real life has a way of doing that.  Child #2 is due in about two months and it’s summer.  Basically it means there’s not a ton of motivation to work on it.  Things are progressing though, albeit slowly.

Additionally it turns out that a few other folks from work are interested in working on SP now too which will be nice.  Misery loves company you know.  Stay tuned for more information - they’ve been pestering me for dyanmips information so I’ve been helping them out with that.

Recently, Cisco added a new “Ask the Expert” thread on NetPro dealing with the CCIE Service Provider lab.  As that’s the track I’m working on I dug into it.  One of the most interesting responses I saw was:

The written exam is not consolidation of former various eight written version, for example, it does not include DSL, Cable, etc technology. Written exam is proposed to certify candidate ability to understand SP theory, protocol and Lab exam certify ability of implementing SP technology to build complicated network to deliver services. So the contents are not exactly aligned. 

I’d like to provide a long term view of SP exam. With SP industry developing, SP exam will include IPV6, IOS-XR and NGN in future, please pay attention to our announcement later. 

I guess in the back of my head I knew that they had to add IOS-XR to the lab but it’s going to make preparing for the lab nearly impossible - who has a spare GSR or CRS sitting around?

If you’ve been to the site before you’ll see it looks different now, and rightly so.  I moved to WordPress from ExpressionEngine to make a few things easier.  I also moved all the content, except the old comments, to WordPress so all the same information is and will be here.

Stay tuned as I’ll be adding the Wiki soon and include all my configs and scripts (and diagrams) for how I have my lab set up for anyone who wants to replicate it.

It’s been a couple of months since I posted.  In the meantime I was busy with life, the universe and everything and (unfortunately) not studying.  I just got everything back online after the hiatus and also started working on my Juniper certifications.  To that end I began working on the Olive which is a Juniper router running on “regular” hardware.  It’s not fully functional but it gets the job done.  I’ve gone a step further though and am trying to get it all running virtually using QEMU (VMWare is not an option here due to fact that the interface drivers in VMWare don’t support multicasting).  So far I’ve gotten everything installed but am fighting to get the routers to “see” each other.  I’ll post as things progress.

So I decided to get a little tricky with my lab setup and failed miserably.

I wanted to reduce the number of systems I have running and figured, “here’s this top-notch server, lets virtualize!” Boy was that a mistake.  Turns out that Xen can’t bridge USB devices to a virtual machine - so after I went and reinstalled everything, it didn’t work anyway!  Then when I tried to revert no flavor of linux would install on the server.  I spent the last 3 days fighting it and finally ripped the 3WARE 8006 card out in frustration and went to software RAID.  Miracle of miracles it worked.  I was not a happy camper this weekend.  20+ hours of work trying to get the dang thing fixed.

Bottom line is, xen is not a good option if you have a lot of USB devices (like I do).  However, the new plan is working beautifully.  I am running dynamips on the dom0 “host” system and have virtual machines for everything else.  Since dynamips doesn’t really write a lot of files or touch things on a system I’m less concerned about running it on the host OS.  That way I don’t annoy my wife by bringing down the wrong box at the wrong time (it’s happened, I have to schedule maintenance windows at my house ).

So more will be coming shortly but I wanted to give a heads up to anyone out there thinking of virtualizing their lab system.  Don’t do it.

I’ve had a number of people ask me via email and on blog comments on what the “best” layout/system is for setting up Dynamips and Dynagen.  I can’t take credit for the answers since I got them from the Dynamips forums but Chris and Greg are pretty clear about it.  The overall “best” system is a multi-processor or multi-core 64-bit system running a 64-bit version of linux.  The reasons are: less overhead, better memory management and better task management.  Since Dynamips is fully multi-threaded (each virtual router uses it’s own thread) the more cores/CPUs you can throw at it the better.

The other big thing is you do not want to run multiple instances of Dynamips on the same server - there’s no need and it actually slows things down since the processes have to communicate via UDP.  You would only run multiple processes if you are on a 32-bit operating system and need to access more than about 3GB of memory.  Anyone on a 64-bit OS (Linux, WinXP, etc) should only run a single dynamips process.

And coming soon - I actually talk about Service Provider stuff rather than Dynamips!  I will add a lot of this information to the wiki on this site for folks looking for information specific to what I’ve worked out on Dynamips.

I have done it.

My lab is now the perfect mix of “real” cisco gear and virtual routers.  I just finished cabling up my server with two USB hubs and 12 USB-Ethernet adapters and wiring them into a pair of 3550 switches.  Then I modified my lab.net file to reflect the new mappings and voila - I have the perfect lab.  I even took an old 2500 terminal server and dropped it into the lab so I didn’t have to virtualize that either.

It’s a little hectic since the ifconfig from my dynamips server now has 13 different ethernet interfaces but it’s worth it.  Yes, yes, I know it’s crazy but eth2 through eth13 are all wired into the lab per the Internetwork Expert SP lab topology.  Pretty darn cool if you ask me.  What’s interesting is that the USB adapters are assigned an ethernet interface on the server and no matter what you do it always gets that specific interface back.  So if you do what I did please make sure and label your adapters as you are plugging them in so if you move them around you still know which is which.  Otherwise you’re going to be in big trouble trying to figure it out.

In case you’re curious the rack contains (from top to bottom in the picture), a 48 port patch panel, Cisco 3524-XL-EN switch, Cisco 2621 router, Cisco 2511 terminal server, two Cisco 3550-24 switches, the dynamips server (1U rackmount), a Xyplex 1640 access server and hidden behind all the ethernet cables is my 2U old-school dual 1.2Ghz AMD server.  With everything powered up (and including a server sitting on the floor that you don’t see) the PDU reports a draw of about 4.7 amps.  If you do all the math and assuming approximately $0.10/kWh (kilowatt hour) it costs me about $41/month to keep everything up and running.

And yes - I know the cabling is a mess right now, I just got everything installed a few hours ago and haven’t dressed the cables yet.  Nor have I bolted the rack down to the slab yet either.

Here’s an interesting tidbit.  If you look at the feature navigator and compare the 12.2S (what is listed on the CCIE SP blueprint) train of code to the 12.0S train you find some major features are missing from 12.2S.  Things like fast reroute, AToM, L2TPv3, etc.  Why is this important?  Well, since 12.2S is on the CCIE SP blueprint they can only test you on features available in that train.  That means that a lot of traffic engineering features, AToM features, etc will not be on the lab unless they update it.  Now all those features are in IOS-XR so if they add a GSR into the mix then it’s a whole new ballgame, but I don’t see Cisco doing that any time soon.  First off it’s nearly impossible to expect people to be able to study if it’s a GSR, second IOS-XR doesn’t really work well yet (speaking from personal experience here).

So take heart valiant reader, it might not be so bad.  Of course I’d prefer they did test you on FRR and such since those are some of the coolest features of MPLS but I’m just going to deal with what they throw at me.

Yes, I could be considered unobservant.  The smarter ones out there will point out that I should have done an IOS comparison a while back but I didn’t.  Not that it matters because if you want to do anything in a service provider network you still need to know things like AToM and FRR since you’ll be using them a lot.  Still, I plan on being a little more focused on the details in the future so I don’t miss things like this.

So my recent foray into getting everything working with the USB adapters got off to a less than auspicious beginning.  I bought a bunch of Zonet adapters which I had read work with linux under Ubuntu 7.04 so I figured they had to work under 7.10.  Wrong.  No matter what I did, what driver I found, whatever it was nothing would recognize those things.  I even tried my CentOS machine and no dice.  I finally had to give up.  Luckily I had only opened a couple of them so I am in the process of doing a RMA with Newegg.

I then ran out to Circuit City and bought a couple Linksys USB200M adapters which cost more than the Zonet adapters ($23 vs about $12) but when I got home and plugged them in - blammo - it just showed up immediately as a new ethernet interface under Ubuntu.  So I ordered a bunch more from Newegg.

My 3550’s should be here Monday or Tuesday so I’ll be able to at least test wire some of this stuff together to see what happens.  I think I’m on the right track now.

I obviously need to spend a lot more time working on MPLS-TE in a Cisco environment. I’m finding I’m forgetting really stupid stuff and then forgetting how to do the debugs on top of it so I can’t even figure out what I’ve done wrong without looking at the “answers”. It’s frustrating really - but to be expected. Most of the MPLS work I do on a day-to-day basis is with Juniper equipment so I’m “new” to the Cisco way of doing things. Oh well, I’ll get it sooner or later.

Key things to remember:

router ospf 1
mpls traffic-eng router-id Loopback0
mpls traffic-eng area 0
log-adjacency-changes
network 150.1.3.3 0.0.0.0 area 0
network 150.1.13.3 0.0.0.0 area 0

That router-id killed me. I had everything else done perfectly but none of my TE tunnels were coming up. I had forgotten to assign the router-id for the MPLS-TE extensions. Duh. Oh well, it works now in the lab. I just have to get better at remembering silly things like this and (more importantly) I must learn my debugs better. Looks like I’ll be going through the class-on-demand a few more times. And taking notes this time.