[WANTED] *nix geeks to save the grid

Linden Lab Production Operations has open positions for Production Operations developers and systems engineers in Australia, Singapore, the United States, and United Kingdom. The Production Operations team is responsible for ensuring that the Second Life grid, the world’s largest collaborative real-time development environment, is up and running.

Linden Lab Operations is a Debian Linux shop. We rely extensively on OSS, and our in-house systems are usually written in Python or PHP. Our team is made up of folks who have been involved in large-scale grid management and site operations for years.

We’re looking for people who can rapidly pinpoint and diagnose network failures, deployment issues, and performance bottlenecks, who can also create tools which will improve grid stability. Production Operations works extensively with the Concierge, System Engineering, Governance, I-world, and Development teams to triage and respond to grid problems; therefore, the ability to communicate effectively with techies and non-techies is critical. The successful candidate will have substantial *nix experience and script-fu, familiarity in managing large system installations, and no fear of complex, dynamic systems.

If this sounds like you, please click here and submit your resume for one of the “Production Operation” postings (Developer or Systems Engineer).

About Storrs Linden

Program Manager at Linden Lab, home to SecondLife. Currently recruiting web developers and production operations engineers in order to ensure that everything about SL gets better.
This entry was posted in Operations and tagged . Bookmark the permalink.

152 Responses to [WANTED] *nix geeks to save the grid

  1. Razrcut Brooks says:

    I already have a career (not computer related) but:

    D3adlycod3c..send in your resume.

    PS:Could someone explain “*nix experience and script-fu, familiarity ” in laymens terms?

  2. Erron Darkstone says:

    I am so happy to read this job offer. With all the problems encountered in SL over the last week or so, it’s comforting to know that every effort is being made to combat the dark side of cyberspace. Personally, I’ve been so frustrated I almost quit SL just to end the constant frustration. My hope is that you get a platoon of qualified applicants.

  3. Argent Stonecutter says:

    Would you have only posted this a month ago.

  4. Jane says:

    Why didn’t you answer our questions about UUID removed from the asset server after a DMCA? The copyright blog was very unsatisfactory, and basically ignored all questions from users.

  5. Agreed, #2. But on the same note I can’t say I didn’t see this coming. I am grateful however, that (albeit without openly explaining exactly why this sudden post, however obvious it may be) they decide to post an open invitation to submit a resume.

    Hopefully they find their Geek-in-shining-armour very soon. 🙂

  6. Alyx Sands says:

    #1: In layman’s terms: If you have to ask this, you should NOT apply! 😉

  7. Jane says:

    http://blog.secondlife.com/2008/04/11/protecting-your-copyrighted-content/

    This blog didn’t post anything new compared to the ones 2 years ago, how long will you keep ignoring our cries for better help against content theft?

  8. Ann Otoole says:

    i’d love to see all that script-fu documentation lol. that should be interesting. yea so if you don’t know what crontab and regex is then don’t apply. interesting trivia: the entire original commerce pipeline microsoft acquired and marketed before conversion to C++ and subsequent incorporation into SQL Server was written in.. drumroll… Python! and it worked.

  9. Michael Fairplay says:

    Great first step. Getting new people in here to stabalize this grid. I hope you can find compitent people to help. Wonderful and welcome step. I can’t say enough good things about this. I hope to see more coherent decisions like this in the future from Second Life.

  10. mj says:

    “*nix experience and script-fu”

    *nix = unix or a unix derivative like solaris, linux, hpux, aix, dynix, minix, 🙂

    script-fu…. a script master…. think like a play on kung-fu.

  11. Wake up in SF please says:

    what SL so desperately needs is an experienced adult who understands public communication

  12. cosa nostra says:

    WELL WELL WELL DONE !!!

    this is the first blog topic who is POSITIVE NEWS !!!! So seems that someone finally read and analyzed the blogs of what happened last 2 weeks !

    WELL WELL WELL DONE !!!

  13. Sheesh.. I was late reading this as I was out buying yet another book on Linux !!! (honestly I was at Barnes & Nobles)

    I will stick to my amature programming and scripting and leave SL in the hands of pro’s.

  14. Razrcut Brooks says:

    Thank you MJ 🙂

  15. U M says:

    Not in Japan? oh please!!!!!!!!!!!!!!!!!

  16. bigmoe whitfield says:

    it’s to bad I dont have the *nix experience needed for this but I am a network IT admin and have already submitted my resume to you over 6 months ago.

  17. Ryu Darragh says:

    *nix = refers to Debian Linux. Fairly common and popular. Written in C and has legs (which is geek speak for has been around long enough to have gone places and developed quite a base of code). Script-Fu = skill at writing scripts in Python or PHP. I’d opine that they also want someone (this would be called a “bonus” skill) good at working with web based stuff.

  18. Greenman Mondegreen says:

    @1, with no disrespect intended, if you have to ask, you need not apply.

    *nix == any/all of the Unix derivitives: AT&T, Berkeley, Solaris, GNU/Linux. *BSD…

    script-fu == ability to use *nix scripting like a martial arts master

    “The script fu that can be described is not the true script fu…”

    Greenman

  19. Tessa Wycliffe says:

    Call me a cynic, but I find it interesting that a massive list of job offers is suddenly being advertised here in the blog. I would venture to say that not many people who read this blog have the know-how and experience LL needs to attack these problems. So what can they hope to accomplish by posting the plea for *nix gurus here? A more productive venue would be monster.com, among others.

    This sounds more like a PR job to me, as in “Look, people, we’re doing something serious about the problem now, looking for fresh meat!” Or, “You sit and whine about the problems, now put up or shut up!”

    Still, it’ll be good for LL to get some fresh programmers in those cushy seats. Let’s hope they want people with visions that may not necessarily gibe with the original one. It’s time LL modified their vision and started looking towards the future, rather than merely putting Scotch tape and paper clips all over the old code.

  20. sirhc deSantis says:

    Well I’d love to apply – but I’m in world. Its my escape. Plus I’m in Canada so bleh

  21. Kerik Rau says:

    @ Tessa Wycliffe

    I think LL™ is trying to get people who are already familiar with the infrastructure in place instead of just pulling any old person off the street. A person already familiar with how things are supposed to be has a leg up in not having to learn the product in addition to settling in to their job.

    The last thing we need are workers who are not “one of the guys” in that they are detached from what you would call the reality of the virtual world.

  22. Lyndon Boucher says:

    You Clowns ….

    ou broke it – now you fix it.

    Looking for a Pro seems to be too late

  23. Andromeda Quonset says:

    I don’t have the experience you guys need for this job, but I do have a Master’s in CSI. I submitted my resume several months ago, and I would be happy with just an internship!

  24. Jazzman says:

    Oh man, you guys are so funny. You slay me.
    I hate to say it but, ROFLMAO.

  25. @19, you’d be surprised the attention a blog can grab.

  26. ActionJackson says:

    Well, it looks like the Lindens are getting bitten by the overall shortage in available Systems Admin talent, especially when it comes to Linux admins.

    Clearly they’re trying to leverage every resource in finding people. Unfortunately, Linden Labs does not have a particularly compelling story to tell as an employer at this point. Their competition does – ranging from large-scale “search” firms to online media (music, video) companies to social networking businesses.

    And given the competition for talent, and the unsettled state of their company, Linden Labs had better focus on employee retention as much if not more than hiring.

  27. Shanti says:

    i was in advertising, PR and can feed hamsters and restick old electrical tape….tape-fu ftw!

    Ok so i don’t qualify….good to see them looking for outside help for the issue though. I can’t help but think of the situation like The Wizard of Oz though – big production lots of flash…. one dude in the basement of some comp lab smoking up and giggling to himself at how many people he can keep online at one time lol Ok probably not true. Probably an attic.

  28. Pansy Peccable says:

    Hahaha, don’t they need qualifications?

    I have been asking, begging basically that LLabs hire someone with ITIL knowledge. Please!

    Service Delivery combinied with Service Management would work wonders.

  29. Dekka Raymaker says:

    The hardest part of this job for those talented to do it will probably be picking their name 🙂

  30. Razrcut Brooks says:

    @ 17 thanks 🙂

    @ 18 ,please re-read my post sir (mam?) I am not applying and have a career I love. Even though your comment seemed snobby at first, you made up for it with the zen-like way you ended the post by intermingling Lao-Tzu’s teaching of the Tao.
    ““The script fu that can be described is not the true script fu…””

    Bravo!

  31. Tessa Wycliffe says:

    Kerik, I’m sorry, but I don’t buy your logic. If that thought were extended throughout the world of software development, no one would be able to hire anyone except from within. And development would remain stagnant, merely maintaining legacy code, rather than bringing in fresh blood with new viewpoints to shake things up a bit.

    And besides, familiarity with the look and feel of SL has little to do with the actual code base. Come on, it’s not all that difficult to have a programmer take a quick look at the running viewer, and play around a bit in world, before attacking the code.

  32. Marianne McCann says:

    @1 – if you have to ask, it’s not the job for you

    @22 – Ya, that will lead to a stable grid, won’t it?

  33. Gregorovich Dragonash says:

    Ridiculous

    All major companies are moving off right now.

  34. Razrcut Brooks says:

    @ 32, mam, (sir?) see post # 30 🙂

  35. Tracy Welles says:

    Funny this is posted right as I’m setting up a debian server and connecting to osgrid.

    You should allow people to work from their location, not travel all over the world helping. People don’t have to work in the data centers for all of what you need in SL. It can be done right here from this chair. I do agree you need the help.

    Tracy Welles

  36. tiny Pechman says:

    Long live Linux, I have been using Computers for over thirty years, and love getting my hands dirty. I have used different Operating Systems over the years, they all have the same problem how do you Multi Task with out creating a bottle neck, one of the main problems is hardware either data coming in too fast or slow! Problem, to have banks of cpu, loads of memory and backup routers, all helped with a good cluster. Any way, Good luck who ever gets the job, they will need it!!

    Regards.

  37. Annabelle Yatsenko says:

    This make me think of the problems eve online had to deal with as it come to a non sharded world. Since there not much of these structures (from my knowledge only second life and eve online) that do exist in the gaming world.

    Eve online seems to have overcomed their problems with the need for speed initiative that started if memory serves back in 2006.

  38. Could it be that LL wishes for what the Users of SL wish for?

    I hope they get their call for help answered, and quickly.

  39. Tracy Welles says:

    I have a low level hardware degree coupled with advanced microprocessor technology experience
    25 years in computers, well since the early zenith systems 8080
    3 years data center experience in networking and cisco
    built custom servers of all kinds
    have layed down more cat 5/6 that most care to do
    can run cisco and various other routers
    am a linux/unix/bsd guru

    And I wouldn’t work for linden and the mess no matter what they paid. You couldn’t pay a person enough to walk into what i’ve witnessed in SL the last month. Unless of course they don’t expect a move, then I might help from here.

    Tracy

  40. Damen Hax says:

    My first impression from the blog title was that SL is heading for something bad, are you hinting towards something?.. Too many randoms?..

    I’m no network guru, but I know my way around enough to see that you simply don’t have the processing power to process the data. How the backlog of data is dealt with (one would assume) needs work also, which would be greatly reduced by the ability to process more information in a shorter period.

    @ 33 All major companies have ‘farsight’.. they don’t just look forwards a day.. or they would never have become major to begin with. This is a preview of what shopping will be in years to come. Anyone with half a brain has a foot in the door.

  41. Pingback: Linden Lab offers a further Australian job opportunity : The Metaverse Journal - Australia’s Virtual World News Service

  42. Trinity Halberstadt says:

    If they actually read the submissions they *do* get (a couple times already), they might find that there are qualified people who are submitting their resumes… 🙂

  43. Razor says:

    how about doing something about content theft?

  44. Tracy Welles says:

    Ryu Darragh Says:
    April 11th, 2008 at 5:57 PM

    *nix = refers to Debian Linux

    Ugh, wrong. nix refers to unix type operating systems period. Not linux or debian.

    Tracy

  45. .......Amazing says:

    WTG LL for taking a step in the right direction! More knowledgeable help is a good thing! You know, it never ceases to amaze me the sheer number of people who will pick apart every single thing that LL tries to do. Yeah, things have been bad this week. Well, for a while. But they are trying to get things stabilized. Give them a break for crying out loud. Bottom line is … if you don’t like it, you don’t have to be here. But please please stop making things miserable for those of us who do. Sheesh.

  46. Uccello Poultry says:

    Thank you, LL™ for recognizing that Grid stability is a priority. Without it I have difficulty providing content. I wish you the best in finding the talent you need. And I hope this notice isn’t because a raft of folks quit.

  47. This is a kinda scary blog considering all the latest glitches and kinks,I don’t know if it’s related but the “bug reporting” has changed and gotten more difficult-this may also be unrelated, but I notice “peak hour” limitations, if SL is experiencing this pattern of growth then perhaps a grid capable of handling 250,000 log ins is in order- do what we all do-keep building-welcome to SL linden Labs!

  48. Tegg B says:

    Gregorovich Dragonash Says: “Ridiculous, All major companies are moving off right now.”
    Hopefully they will take their humourless mono-drones with them.

  49. Tracy Welles says:

    I just cannot picture Linden offering anything of value for the job. That job should start at no less than 65K per year and up with a “fairly okay” resume. With a good resume plan on 100K.

    Then take the fact that they offered 10USD for content creators to help out in SL and I have a feeling they have no clue what they are going to pay for anyone that knows anything.

    Tracy

  50. Miguel says:

    Hoooah! Finally i’ll get my resume looked over! (not rly)…
    In essence … ty Lindens (©® TM) sry not sure what to use… for recognizing you don’t have a clue… and dumping a “there’s the ball grumpy resis, now come help if you know better”…

    /me sighs and goes back to osgrid…

  51. This post needs a “Linden Labs needs you!” Uncle Philip poster.

  52. Annabelle Yatsenko says:

    I dont know if I should add anything yet I will.

    I’ve seened this before. The same kind of posts that depicts apocalyptic end of the world scenarios I’ve seened it elsewhere. For a fact with this kind of challenges player base will have to be at some point a monthlee fee subscription to pay for the upcomming upgrades to the cluster of second life.

    http://www.superssd.com/success/ccpgames.htm

    http://business.pcauthority.com.au/news/eve-online-sets-gaming-supercomputer-cluster-record.aspx

    http://softwaremaven.innerbrane.com/2007/12/python-versus-erlang-for-mmog.html

    Since there is only this server that is similar to second life I use this to highlight that there is solutions and second life devs team will probably find their solutions also. Yet I have a genuine feeling the solutions will be costly in terms of harware and recoding-bugfixing and probably optimising the cluster’s sofware language to speed up exchanges between client and server side to reduce lag and or random coding errors leading to node-server-cluster-server crashes.

    One thing I dont really get is that second life is not using cable internet over dial-up conexion. On the other hand I am not an expert on this matter and dont wish to be one.

  53. Lee Ponzu says:

    What happens when you add more man power to a late project?

    It gets later.

  54. CheerGirl says:

    get paid in $US or Linden Dollars?

  55. Isn’t it funny all of the sudden you guys want to hire people when you know its too late to save SL. These people should have been hired last year before all the problems came….

  56. Every time I see this type of message I think they fired someone. Just a hunch, no facts to back this up but that is usually how the business world works.

  57. RC Paderborn says:

    Tracy, I think you are vastly underestimating the salary for this job. That is if the person has to relocate to the Bay Area. For the experience that LL is looking for, $65K would cut it say here in Las Vegas, but not Denver and certainly not in SF.

    I know a 22 year old gal who is a glorified secretary (administrative assistant) in Berkley who makes $60,000 a year and has only been on the job a year. Certainly an experienced large system *nix guru would get more than that.

    By way of comparison, her tiny loft apartment near Berkley is $2,000 a month. (it’s some “sliver” of a town that borders Berkley. Can’t think of the name at the moment.)

  58. sophia Yates says:

    Well I really really hope this all gets straightned out..i couldnt even get to the short pier to hurl myself over. Crashed before i did and couldnt say goodbye to my dolphins who r still hanging in mid air and my elevators r wiring away and my potery wheel stoppped and the texutres look like mud now and my car doenst work. And well please do something. Its becoming borderline insane even logging in. Why bother. Its a shame SL was alot of fun.

  59. RC Paderborn says:

    Tracy, sorry… just reread your post. Yes $100K for a good resume would be a good starting point.

  60. Tabitha Oxide says:

    “When the oxygen masks fall from the compartment, please place the plastic cup snuggly over your mouth and breath deeply…..”

    “Passengers whom have brought their own parachutes are encouraged to exit through the rear of the plane at their discretion. Thank you for flying Linden Lag.”

  61. Flurry Splash says:

    What we really need is someone that understands the binary language of moisture vaporators.

  62. Will we be seeing the post….

    Linden Labs needs CEO.. If you have any brains please apply..

    What about customer service?? Your not hiring for that???????

  63. Prospero Linden says:

    Could it be that LL wishes for what the Users of SL wish for?

    Ding! We want a stable, happy, functional, usable Second Life. It’s the goal we all strive for.

    Answering some other questions : this is not a “sudden” job posting. In fact, the Production Operations Engineer position has been listed on http://lindenlab.com/employment page for quite some time. At least since last June, which is when I applied for it….

    There are already several Lindens (including myself) in this group. Part of the reason we want somebody in Austrailia is that we want to have production ops people available around the clock while being up at their natural time. (Mind you, I’m in the Central timezone — Nashville, TN– but some of those on the west coast have a natural time that’s later than me, so the sun is of course a part of the picture.) Right now, we have the least coverage in this group at times that would be most convenient for people in Austrailia and Singapore, but of course we want any and all good people to apply.

    We’re not looking for One Uber Geek to Fix Everything. Rather, we’re looking to complete the team that is already in place so that we’ll have more complete coverage in this group. That will take some pressure off of a number of other groups that will be able to re-focus on designing the infrastructure, designing the code, and providing customer support.

    You do not have to understand the binary language of moister vaporators, because we’ve already written a Python script to do the translation for us.

  64. Jenny Carlos says:

    Debian at least they use a good flavor of Linux lol.
    I saw someone say something about other “Generic” “Crap” versions of Linux. I surely hope that Linden Labs isnt using some of those listed from that post no wonder we have the problems lol.
    Then again it could have been windows based 🙂 rolls on the floor lol.

    Id have to say that finding “Good” Qualified Linux techs is going to be a job all its own.

    You cant have just any Linux tech just “Using” Linden Systems let alone writing code lol.

    Anyway good luck to Linden Labs and maybe expanding on all of the areas of second life team is a good idea.

    Maybe you can fix the horrible lockups and memory leaks and well lag , lag and yeah umm did I say Lag ?

  65. Jenny Carlos says:

    Prospero Linden , Well said . And all the luck to you guys.
    Like I said expanding on any sections of Linden Labs is something im sure we are all happy about as it benifits us all in the long run.

  66. Zeno++ says:

    Looking over all this I just have to comment:

    I come from a financial background where I’m use to hundreds of computers with hundreds of processes handling thousands of transactions at a few micro seconds a piece. I know SL has far more servers then we did. But it looks like we had much more extreme SLAs. Temporary shutdowns could bring on a recession!

    I don’t know where Python came up but I have to laugh! The financial world would be in ruins! There were experimentation but we found it invited hacking mentality. Almost all Python projects looked good at first but didn’t scale. Its now only used for in-house supporting tools…

    C/C++, Infiniband, shared memory, numa, assembler, asn1, ucm, process, are words that should be brought up in your search. More skilled and creative project management by people who do it because they like to get things done and are very technical! Creative people are what drive a company to success. Those are the things that have made development successful in my life.

    Hint, the financial businesses are crumbling and more and more people in NYC are getting replaced by outsourcing. Look there for the hard core C++ gurus you need! The rest are writing video games!

  67. Amaterasu Cinquetti says:

    Prospero:
    If you ever wanted Australian Customer Support people to work during natural Australian hours I would apply in an instant. That is another aspect that is very sadly lacking in SL.

    good luck with your hunting 🙂

  68. Brieanne Bomazi says:

    *shakes her head with a grin* ok folks,….Pay attention…some of the BEST lindens that curently work at LL came from job postings just like these… 4+ years in sl (yeah, so this is an alt, sue me) …the grid has had MAJOR ups and downs..and everytime there is any hint of change people get all negitive, and gripe and complain… if you cant be part of the solution…shut up griping about the problems. I personally am not qualified for such job… but i am sure there are MANY that are….kudo’s LL for again making an effort to make the players part of the solution.

    May the script-fu’s be strong..and the grid get stable.

    Brie 🙂

  69. Cherry Czervik says:

    @11

    I should be sending in my CV … well let’s say resume.

  70. JayR Cela says:

    Is about time you took a serious approach to hiring qualified people to run the Servers and such for the system you have been building for the past 8 ~ 9 years :_)
    Wish I could help 😦
    May I wish you all the best luck in the world / and Hang in there!! Things cant possibly get much worse then they have been during the past several months.

    JayR Cela :_)

  71. Sovery Short says:

    Hmmm. I have some fairly large band aids….Couldn’t possibly be any worse than the current ‘FIXES’ being applied. At least mine are waterproof.
    @67 Oh to have some real help personel around instead of all this cloak and dagger hidden help stuff.. But.. first things first I guess… stability in both the servers and the programme will be a great start. *rolls both eyes and leaves*

  72. Calinacase Whiteberry says:

    Ann Otoole 8:
    i’d love to see all that script-fu documentation lol. that should be interesting.

    what would be interesting would be to see if all the script-fu wizards @ LL actually documented their work. For some inexplicable reason, I doubt they do. Maybe it’s the old bugs that generally pop up a few rc releases later.

  73. Linux servers, good :))))
    Well, At least it won’t be dealing with Windows issues. I hope.

  74. U M says:

    DING!!!!!!!!!!!!!!! its who you know that get you in DING!!!!!!!!!!!!!!!!!!!!

  75. @1 Well *nix means any form of Unix, Linux etc, and script-fu. Well, it means it’s all fubar to start with.

    I think it’s already far too late to save the grid now though, meltdown is unavoidable at this rate.

    @63 Prospero, “Ding! We want a stable, happy, functional, usable Second Life. It’s the goal we all strive for.”

    Really? We have been saying that for a year, get the core fixed and stop adding on shinies that overload the system more. What do you do? Ignore us and add shiny after shiny. Funny how the latest 4 week long catastrophe, comes after the server code to add Windlimp and Havok (sure is creating that), and the total grid fallover since the viewer for that was made mandatory…

    Still, I’ll bet the sim crash stats and system downtime figures will look good tis month, never mind everything else has fallen over and you can’t actually do anything at all inworld. But then we’re not supposed to I guess, we are just supposed to stare at pretty skies and go woot, wtg LL…

  76. Datapanic says:

    If you guys ever switch to Solaris or a Berkeley derivative and dump MySQL for something more robust such as, well, Oracle, such a job, providing it allows for full time telecommuting and something over, well, 150K USD per year would be interesting. Your startup did not plan ahead, and 5 years later, you find that your platform cannot handle the load.

    I would love to work for you, but I do not stick my fingers into other people’s poo nor wear there underpants!

    Good luck!

  77. Anyway, glad you guys finally noticed what we have all been suffering. And there was me thinking it had all been written in Ruby Fell Off The Rails…

  78. wuvme karuna says:

    YAY!

    i hope a bunch of nerds apply and team up to help save second life! or at least make it more stable!!

  79. Bie Doobie says:

    @64 ¿ why lunix why not full blown solaris using and instable os could be the cause of stability proplems from the start this server that server blah blah unix is industry proven itself for years.

  80. Hiroaki Raymaker says:

    Heh, well, I applied. Should be interesting to see if they nibble. @76: I understand your reservations, but a poorly admined system can crash all over the place regardless of the OS – I’ve seen Solaris and various BSDs crash just as nicely as any other OS. I also have Linux boxes on production (not network-facing) clusters that have been up for more than 500 days. Were it my choice, they’d migrate the database backend to PostgreSQL for the database assuming they want to remain with a Free choice, but any change like that’s going to introduce instability as they migrate the entire stack(s) to a new database with different SQL ‘enhancements’. Oracle’s great for such a large database app, and it’s fast (it’d better be, with that huge chunk of shared memory it allocates for itself) on decent hardware, but it’s definitely possible to scale other databases. If one’s not willing to get their hands dirty, well…

  81. Ms Z says:

    To comment #1 Razrcut Brooks. They are looking for geeks who actually know those terms. If someone has to ask what they are in layman’s terms guess what? They don’t qualify.

    To the Linden’s: Please Can we get a coupon for ten dollars off or more to buy a pair of Ray Bands? That way we can fully “enjoy” Dazzle? =p

    (Sorry but there was NOTHING wrong with the old look and was easy on the eyes. Now being on Dazzle I get a headache after an hour. So I guess I wouldn’t be on as much as I use to…buying Lindens and spending them.)

    Oh and if you want to get people to buy more land how about lowering the price of the monthly tier fee for those who have been your loyal and beyond patient customers? That way land owners can actually break even when it comes to renting out our prims.

  82. Clutch says:

    well “save the grid” sounds appropriate…could you people fire the morons that are “destroying” the grid?…sl has progressively gotten worse since havok 4…so much for improvement…it was much more stable and fun last year and every so called upgrade turns into a downgrade in performance. step up and spend money on people that know what they are doing and stop hiring college dropouts for minimum wage…your world is dying in front of your eyes and no one in power seems to care.

  83. Josh Panther says:

    *laughs* I love how I keep seeing people saying ‘Last year’ was better or more stable…clearly you havent been around for very long.. SL has never truely had much stablity, there was always issues witht he grid for one reason or another, and various other problems… just ask Tabitha Oxide who keep me laughing with every post she makes on these forums…

    Annabelle Yatsenko, Thank you for the post, I hope that a Linden reviews it. Your post was about one of the most informitive postings, I have yet to read on this thread.

  84. Demetra Soderstrom says:

    Wow! I’m surprised to see my small island country is wanted up there but unfortunately I’m still in the process of studying a computer related course in college. I would actually love to help LL and work for them.

    *sighs* However this is a great opportunity!

  85. Masami Kuramoto says:

    #80 Bie Doobie Says:
    “why lunix why not full blown solaris using and instable os could be the cause of”

    lol. Linux unstable? You must be on crack.

  86. Audrena says:

    I’ve been trying for over an hour just to log in again. I certainly do hope they find someone that can keep it all working for a change. it seems to just get worse and worse as time goes on.

  87. Coventina Dalgleish says:

    Well you certainly need some help as obviated by the data base problems again tonight, with low concurrence. LETS Hope you find some qualified people who can take a jaundiced look at the way you have this thing interwoven as it obviously does not function in it’s current state. This is a daily excersize in futility.

  88. shiro says:

    if they are asking this in a blog post then they are already beyond finding someone that will work. perhaps they just dont know what they need or their offer is poor but needless to say if you are using a blog post to find resumes you are already underwater and no flotation device is necessary.

    LL could do to get their heads out of there collective asses and make the grid more stable. it honestly isnt that difficult and if you buy their excuses over the past few months then you also think we are spreading democracy in Iraq.

    GL LL but find a guru or just give up.

    (btw, friday once again and nothing is working right. big surprise)

  89. Clutch says:

    Josh Panther Says “clearly you havent been around for very long”

    clearly you assume that you know everything *laughs*

    and thanks for informing us that the grid has never been stable…issues with the grid too? really?
    im sure alot of people had no idea…rofl

  90. LoLo says:

    Hire some extra people like Sidewinder to give answers on the blog. The copyright blog was a complete disaster. So many questions asked, and no proper answers. Fire LauraP (or let her do other work which is not customer service related) and hire someone who gives a proper answer to our questions.

  91. U M says:

    laughs. its been over 2 1/2 years now things have been a total mess……..The only real decent person in the lindens these days is Sidewinder. Hire more of these days and less of the rah rah and ones that can`t post a hyper lin correctly.

  92. None says:

    First get servers that can hold more then 50,000 users. Don’t you see that when you hit 50k your stuff crashes. Your servers can’t handle it.

    Second advertising for people to get a job from a blog is horrible. Try Craiglist or some other service since you want people in certain locations.

    Third database management would be nice. I run servers and have never had to tell them to stop, hold up, and restart them. You at LL must be caching the holy hell out of those things. Because you shouldn’t have to restart the hell out of them everyday. And explain how taking profiles and little services helps? FLMAO! I read that and just thought… WTH?! How is that supposed to help? Running separate DB’s for every service is just poor management. One DB… Unless you’re running the horrible MSQL it shouldn’t crash as hard as this one always seems to… but the magic number is 50k… You hit that and it just chokes.

    And what about us who pay for this service? Where is the love for the land and island owners… Paying for a full month of service when just a couple days ago the grid was down for what… Over 24 hours? People who were in could use it and then kicked out way later. Businesses lost money, and this constant… HEY DON’T MAKE ANY L$ TRANSACTIONS is horrible. You’re telling people not to spend money and in turn making those of us who pay lose money. *shaking my head in disgust.* I know quite a few people who have went to IMVU and tell me it’s wonderfully stable. I’m wondering how good it is now, because if I’m paying for something I expect to be able to use it when I want to.

    Why don’t you just cut peak downtime BS by limiting the number of free accounts that can get on. If you’re not a paying member you have to get in a queue and wait to see if you can get on. Because I don’t mind paying for something fun if it works, and if others want to complain because they can’t get on whenever they want for free who cares? Because land/island/space owners, and paid members like me are paying and they’re just freeloading.

  93. Silk Aeon says:

    (/me pointing again login ptoblems)
    Have you enought steal a money from residents? There are must be enought to hire proffesionals and install normal operating system.

  94. shiro says:

    i dont get it. during the week do you have people with towels standing right next to the DB and other servers to (barely) keep them up? because every weekend its a domino effect of complete and utter calamity?

  95. Ken March says:

    Is there opportunity in china mainland? hope i can do sth for LL.

  96. Nibb Tardis says:

    Sounds desperate.

    The Lab also needs to hire
    – decent PR people to save the community,
    – some marketing folks to save the product appeal,
    – a customer support manager to save customer satisfaction,
    – an entire QE team to save software quality and processes

    Oh, and I hear the CTO position has been open since december, and I wonder how the search for a new CEO is going ?

    LL has good tech folks, it’s the management and (lack of) business processes that is really running them into the ground.

  97. U M says:

    WOW someone with common sence here FINALLY!……..”97 Nibb Tardis Says:

    April 12th, 2008 at 12:41 AM
    Sounds desperate.

    The Lab also needs to hire
    – decent PR people to save the community,
    – some marketing folks to save the product appeal,
    – a customer support manager to save customer satisfaction,
    – an entire QE team to save software quality and processes

    Oh, and I hear the CTO position has been open since december, and I wonder how the search for a new CEO is going ?

    LL has good tech folks, it’s the management and (lack of) business processes that is really running them into the ground.”

    Their PR department been nothing but a joke. Oh they tried but they landup with some pretty crappy and unprofessional people. Why? because they don`t hire people people. Its all done inhouse. Now if you track the past 2 1/2 years worth of PR stunts LL “TRIED i you this worked lightly” it just blowup in their faces. Gaming was the on The ( SECOND LIFE FRONT PAGE )worse nightmare LL had.” remeber the HEY I WON AN YOU CAN WIN BIG $000,000.00L in Poker …..? everyone knew it was a sentup. The person never won the money they payed the owner that amount and ran back to a newbie club to say YOU TOO CAN WIN BIG. YEA RIGHT BIG BS” These types of PR 110 class ways of doing PR is just childish and totally sad. These people have college degrees? Community rep are another joke. but can you blame LL? They don`t have much to choose from and rarely hire outside.

  98. Sidewinder Fan says:

    We need more Sidewinders

  99. U M says:

    agreed………more meat and less fat please

  100. Danel Kurosawa says:

    Please Lindens…Please…try to work ONLY in the STABILITY of the grid. The last week was terrible for all of us who have business here in SL. I am loosing big amounts of money and i need these money to survive in RL. This is not a joke folks
    Please

  101. Ms Z says:

    Would have to agree with Danel. In nearly two years of being here I’ve never seen so many transaction issues in the last two months. It hurts all the builders of sl and it kills things for those who love to shop. I don’t think we all care for being distracted with your Logo, Dazzle, voice and other bits of candy, bells and whistles. We want stability before the addition of new things. And we want the new additions to be tested out more before they are added to the main grid.

  102. Plum Planer says:

    And it would be so nice if you Lindes gave us a prove you actualy read these comments too, I now get the idea you are not listening to any of us but only to the big companies that try to get into SL

  103. Jayden B says:

    /me applies and hopes.

    *nix geek in Australia

  104. Blinders Off says:

    /me finds it interesting the job post uses colloquialisms such as *nix and script-fu.

    What are you looking for, a professional systems manager or a teenage hacker?

    I think I may see part of the problem with LL stability issues…

  105. Blinders Off says:

    @ 97 Nibb: agree 110%

  106. Masami Kuramoto says:

    LL, you need more developers? Open the server source!

    It worked quite well for the viewer, didn’t it?

  107. Pepper Haas says:

    I sincerely hope you hire someone who knows that it’s a major crime for any software publisher to change the GUI, so I don’t ever have to put up with having to adapt to the useless changes LL has made to it four times already. The IM/friends/groups window is a sorry mess.

  108. bobbyb30 zohari says:

    Wow LL, just wow.

    I realy feel bad for whoever is “lucky” enough to get selected to be a Linden.

  109. riona says:

    Call the A-Team…
    In your case this is your only solution

  110. Captain Noarlunga says:

    All that this post does is to confirm that the current staff (with a very few exceptions like Sidewinder), are incompetent and incapable of analysing and fixing the current problems. Now we have a big problem…..when these new people apply…..the idiots will be deciding who to hire ! Its like the asylum inmates deciding on who the shrinks will be! LMAO And…like others have commented…..LL needs people who can manage the whole thing……its no good having people in the engine room if the captain has the boat on the beach! Lets face it, this is too little too late. SL is in meltdown, this is a PR job to keep people happy until the IPO. After the IPO the chiefs will be mighty rich and we will be the suckers who made it possible for them.

  111. Captain Noarlunga says:

    LOL……my post is awaiting moderation…must have hit too near the mark.

  112. Captain Noarlunga says:

    We appreciate your comments….but dont tell the truth……so typical!

  113. Captain Noarlunga says:

    think if you use IPO in your post it is a moderation keyword!

  114. Hulk Ah says:

    Nice post,

    WANTED : a linden that will handle dmca’s better than just removing one location but instead removing things from the database like they should

  115. Captain Noarlunga says:

    All that this post does is to confirm that the current staff (with a very few exceptions), are incompetent and incapable of analysing and fixing the current problems. Now we have a big problem…..when these new people apply…..the iincompetents will be deciding who to hire ! Its like the asylum inmates deciding on who the shrinks will be! LMAO And…like others have commented…..LL needs people who can manage the whole thing……its no good having people in the engine room if the captain has the boat on the beach! Lets face it, this is too little too late.

  116. Ryu Darragh says:

    #64 Jenny. Windows is no worse, nor better than any flavor of Linux. It’s not the OS, it’s what apps are available, what the capabilities and experience for each of those and what has been used for the purpose in the past. When Mac was the only platform that had publishing and art (photoshop type stuff) programs, like Quark, it was *the* machine (and OS) to use. When the Amiga was the only platform with decent graphics and video editing hardware and apps (Toaster), it was *the* machine to use. Since most server platforms used Linux, because it’s such a stripped down and simplified OS (you can run it on the CPU in a watch, for crying out loud), it was chosen as *the* OS. I use WinXP because all the apps I need to do embedded processor work run on it. You can find apps like the ones I use that *do* run on Linux, but they are hideously expensive. Ask the guys at Pixar what a high end workstation and all it’s software costs.

    No, I work with Linux (in its RTOS flavours) all the time and they are what I use for a lot of my work, but, they are *not* very usable for the average home user. WinXP (and fully shelled and protected and pre-configured Linux) are. Of course, in that sense, Linux that the average user can tolerate is just as unstable and prone to problems and exploits as WinXP.. gee, wonder why that is ?

    So, folks, *please* stop with the “Linux is superior to WinXP” hoo-ra. It won’t solve the problems here. It’s like complaining that Beta is better then VHS. Once VHS was the major player, you had to live with it or live with nothing at all. Look at the Blu-Ray / HD-DVD fight. Yes, Blu-Ray (it’s a deliberate mispelling) is superior and it won, but what good did the fight do the industry ? It nearly never happened at all because nobody bought anything 😛

  117. WiLLuMPJuH Gausman says:

    Let’s see here.. is this Debian being run on those servers still using old stuff instead of what i am using now ? And will i be able to use Dazzle which is compiled against an outdated GTK ? .. and wait .. one big advantage .. development is that slow i will never need to upgrade those servers like is never done now .. just restart a lotta services that are overloaded , i suppose…

    Compiled against GTK version 2.4.14
    Running against GTK version 2.12.9
    Compiled against SDL 1.2.5
    Running against SDL 1.2.12

    Forget it .. i’m consider myself overqualified.. 😛

  118. WiLLuMPJuH Gausman says:

    Or underqualified considering my speeling .. :S

  119. Aya Pelous says:

    Linux is NOT better at all

  120. Malachi Petunia says:

    I would urge caution on the part of anyone seriously considering joining an operations team with such a poor track record.

    Even though the blog title is probably intended to be cute, asking someone to “save the grid” that they’ve not been able to do for years is a daunting proposition. That they seem to rely upon customer reports – even today – to “detect” faults indicates that they may not have the automation in place to detect major faults. If this is so, a qualified candidate may find themselves fighting an uphill battle to impose common NOC practices on a firm that hasn’t seen fit to use them.

    To use a sports analogy (even though I don’t know sports), if you are recruited to a team with a 0-48 record, you might be wary of just what is expected of you. Nevertheless I tend to agree with the comment above that this is merely a PR statement for a position that has been unfilled for about a year. I’ve also heard that despite these solicitations, LL is quite poor at responding to even qualified candidates.

    I’m not saying “don’t apply” I’m just recommending that you ask a lot of questions should you interview.

  121. hey dont worry everyone, the requirements may sound tough, but in the end all you need to be able to do is spell out the world “resolved” and post it on the blog from time to time. Work at LL really doesn’t involve all that much more then that.

  122. Kugel says:

    Screw all this lot… they need to do a SERIOUS post-mortem on all the TOTAL fook-ups that have occured these past weeks with their infrastructure.

    AND stop useing smart-assed words like “Happy servers” and “Unpleasentness” – cos I dont know about anyone else… but it riles the hell outta ME !!!

  123. Jules Joffe says:

    #105 Masami, agreed. Opening the server source would yield public review and patches from a community far larger than the paid-for people being hired here. Why not to prioritize clearing that path?

  124. So is actually anything at all working this weekend? Why don’t they just close the grid for a coupole of hours, and reboot every machine. I mean, it’s noit as if we can do anything in world anyway, with transactions failing, region crossing, database borks, and everything else.

    Better yet, roll back server and viewer code to 1.18, which was working pretty fine until they added in the borked Windlimp and Havoc..

  125. Escort DeFarge says:

    @118 Don’t underestimate the issue! A large enterprise system worth its salt would be expected to manage approx 20k simultaneous logins. LL have to manage three to fives times that (60-100k logins) for SL. So the problem is exponentially harder. Frankly it’s a great technical achievement that the grid is usually operational… but you are right that despite all that the grid should/must be stabilized.

    Employing firefighters is a good idea, but I don’t think it will solve the issue. I’m assuming that this call for good technical staff is not the strategic plan (which incorporates such projects as havok4, mono, etc).

    @80 Personally, I too hope that strategic plan also includes migration away from MySQL to Oracle or Postgres for the reasons stated here…
    http://jira.secondlife.com/browse/SVC-1149

  126. Wayne Letov says:

    This is an internal LL email;
    Tisk, tisk, hey dudes and dudettes, them there databasers is actin up agains and beings mighty unpleasant. Someone make a funny post, tellin thems people [who’s likes to calls themselves customers- but we’s all knowd theys be sucka’s], and that we knows what we’s a doin and workin on it as best we cans. They’s fall for that stupid BS every timed this there system acts up. Besides, what er they goin’s to do? We’s has their moneys, and we don’t a gives any of them there refundy thingies. We’s just gotta fools them there sucka’s into thinkin that we’s cared abots them, so’s they keep givings uses theirs moneys.

  127. Ravanne Sullivan says:

    One suggestion. Make sure whoever you get NEVER, EVER, UNDER ANY CIRCUMSTANCE uses the word RESOLVED in a blog posting.

  128. Shanie says:

    Yes, it would be nice to be able to purchase linden without being ripped off. Even lindex is broken-purchased linden yesterday that was never received-but money was still taken from paypal! A warning to all-think twice before making any $L transactions!
    if u can’t keep the website stable there is no hope for the grid-too bad

  129. U M says:

    frankly speaking LL doesnt care. If you do thery is very little hat can be done………Loves hows this blog hold people post without any reason

  130. Pingback: I think we are at End Days, really . . . - Page 3 - SLUniverse Forums

  131. Captain Noarlunga says:

    Guess this shows how things are going…a blog post that allows comments and it isnt full. People have given up now. This is Phil & Jacks greatest achievement……..creating a sense of hopelessness. What a legacy.

  132. Tabitha Oxide says:

    Honestly.. I stand by just watching the news. I think that sooner or later with all the millions of people who are involved here as well as other MMO style worlds and platforms does something psychotically drastic. Some one or a group who have been pushed so far with the compounding issues of world economy, leaders, deception and now the deterioration of a virtual world they hide in because they had some control of the environment.. lose it. Im surprised it hasnt happend yet to be honest that someone or a group of lunatics havent drivin a truck of fertilizer right to the front doors of LL in california and blown the place to rubble to satisfy their own psychotic need of some resolution.

    To me, I would be scared as hell working for LL in california or any location for that matter with the seemingly millions of users and investors who are completely unhappy and or pissed off beyond normal human behavioral control. I would be living in fear that some lunatic or luntatics arent concoting a plan to attempt to be martyrs in their own vision by bringing this mess to a final resolution by means of terroristic tactics. Sure thats completely insane but we live among completely insane people who do such things for far less stressful experiences than this. Even in WOW (world of warcraft) the news has articles of users who have killed others over just in-world Items.

    It would scare the hell out of me to be anywhere near or associated with Linden Labs at this point. A few months of problems is understandable but years of issues making the world unusable? Policy changes and cost reversals made in an instant? I knew people who made thousand running casinos, then one day it was over. Their worlds came to a sudden halt and lives changed. I was surprised some act of agression wasnt publicized then by some group of psychotics seeking revenge. The world is full of crazies that do such things with little provocation. Add to that the distresses of the economy, world leaders and the fact that surely some found SL as a place to bury themselves and hide in an environment they could control but no longer can? Its a ticking time bomb and someones gonna try and reach martyrdom by doing something insane that will make headlines across the globe.

    I would seriously be very concerned of such actions and do whatever it takes to settle the building pressure of negative public opinion as fast as possible or just take it down (offline) before real people get killed.

    The real world is anything but stable. I read blog posts that are filled with very very upset users. Some new some veteran. Some who just hate the lag and some who run businesses here that they depend on for their livelyhood that have suffered greatly do to actions taken by the internal company string pullers.

    If it was a world of 30,000 I wouldn’t be so concerned but here you have millions of people of all walks of life across the globe. Surely there are many who take this all WAY to seriously and could be pushed easily over the edge to do something drastic. I would just take it offline for the REAL safety of those who are involved as lunatics dont care about life and the law, they get shoved so much into a corner that they lash back often taking innocents with them.

    So here I sit.. Unable to login, Unable to use the platform I have been a part of for over 2 years coping with my personal frustraitions in a stable and humanely manner while knowing that there are people in their basements who are not so stable smashing keyboards, swearing at monitors slowly being pushed beyond their control limits unable to just logout and quit the world because they can’t cope. These are the same people that re-enter their colleges and kill teachers and students because someone picked on them not lost tens of thousands in a virtual world.

    My advise, fix it or pull it till its fixed. To many crazies on this planet. Sooner or later someones gonna crack and innocent lives will be lost its inevitable.

  133. Tabitha Oxide says:

    After some thought, taking offline may in fact just be the match that lights that flame as I know sooo many have so much invested and operating. Pulling it isnt a solution.

    Interesting that they are stuck in this situation. The only remedy is to fix it as soon as possible or at least stabilize it to an acceptable functioning state.

  134. Olish Newman says:

    @105 : You are totally right ! Open the server source ! Anyway, LL fears about data security and theft… Or may be do they fear about their business to go down…

    There is already an open source metaverse project : OpenSimulator. And it’s going ahead at a good pace…

    OpenSim is freedom.
    Viva OpenSim !

  135. Pingback: Second Life web resources for April 4th 2008 through April 12th 2008 | VintFalken.com

  136. SGI Cool Breeze says:

    Perhaps the simplest solution would be for you guys/gals @ 2ND Life (TM) just to invest in some better air conditioners for you overheated servers.

    Nope, I don’t want the job, but thanks for asking.

  137. Clang Bailey says:

    @127 – What?

  138. Lesley Grose says:

    linden labs hire a lawyer for the DMCAS they suck

  139. Jane says:

    Why didn’t you answer our questions about UUID removed from the asset server after a DMCA? The copyright blog was very unsatisfactory, and basically ignored all questions from users.

  140. Resse Brane says:

    @137
    Linden Labs doesn’t cares about texture theft and has even made prim copiers legal. So now all big shops made their item NO MODIFY. Yay! No more editing my belts, hoes or hairs to fit. GREAT.

    Onr more reason to run to the first competitor of Secondlife that DOES take our complaints seriously.

  141. All those years learning the binary language of moister vaporators….. waisted.

  142. Meta Starostin says:

    Sorry, but after 30 years in the crappy IT Industry, I have decided to run extremely quickly in the opposite direction.

    Now back at University beginning of this year – humanities related degree.

  143. Chris says:

    Chances are that if you don’t know what it is you don’t have it. *NIX means UNIX-like systems, such as Linux. They want you to be familiar with Debian GNU/Linux and it’s workings (as well as BASH)

  144. Meta Starostin says:

    Linden Labs is looking for developers and systems engineers who can rapidly pinpoint and diagnose network failures, deployment issues, and performance bottlenecks, who can also create tools which will improve grid stability.

    If you don’t have a degree in Computer Science, or equivalent, and don’t have relevant experience in large scale server installations found corporate data centers or software development experience in Python or PHP languages, you need not apply.

    There are literally thousands of flavours of Linux, one of which is Debian. All flavours have one thing in common – the Linux kernel which was ‘developed by Linus Torvalds in 1991 and derives much of its basic design from principles established in Unix during the 1970s and 1980s.

    Even if I was still in the industry I would not apply. My Linux network experience is limited to Red Hat and SuSE – Enterprise as well as desktop versions.

    HR in many organisations are completely erroneous and incompetent in their screening of candidates and only limit selection of candidates who possess only one flavour of Linux when they are all almost identical. This precludes very suitable candidates.

    Considering the length of time Unix has been around (1968) and Linux (1991) why do they need to go offshore?

    Offshore to Bangalore – plenty of cheap labour there.

  145. SLick says:

    I am wondering when people are going to finally understand that that virtual worlds will never work flawless. There is always going to be a problem and some things aren’t fixable. I personally have the up most respect for LL and the hard work they put into SecondLife. At least they are trying to fix there issues. In my opionion they are doing a damn good job. SO kudos to you lindens out there, and to the people that aren’t happy no one is holding a gun to your head go play something else. If you are here to make money to support you in real life then you are here for the wrong reasons.

  146. Renee Faulds says:

    Here it is Sunday and services are down AGAIN !!!

    Are there any business people that have survided GOING ON 3 WEEKS

    of down time and crapped out services ???????

    Phil YOU LIED to us with your last blog post the first of the year !!!!!

    PAY MY TIERS LINDEN LABS !!!!!! YOU OWE IT TO US !!!!!!

    Renee Faulds (getting broker and broker)

  147. Raindancer Raymaker says:

    stand back for a heap of tips blogs to wipe all the mess off the blog….great weekend…………….

  148. Kugel says:

    And as someone said earlier…. there is so much apathy… this lone open-comment blog still isnt full of ranters and ravers.

    I dont care what the excuses are any more, or where each problem lays… SL has become a 5fps, TP failing, money failing, logon failing joke.

  149. nomoresecrets says:

    SLick AND NO ONE NEED TO PAY U INGAME; SLC

  150. Captain Noarlunga says:

    Maybe they should be advertising for a lawyer to help fend off all the claims coming their way for theft. 🙂

  151. Anny Helsinki says:

    ORE FIND THE PEOPLE THEY BUYES ACCOUNTS AND MUCH LAND WITH

  152. Anny Helsinki says:

    Pingpong hehehehe

Comments are closed.