Jun 22 2016

xmlrpc attacks and how to handle them

Category: TechnicalIuliana @ 23:05

Here I am managing my own blog. Last week the blog crashed. Which made no sense because everything was brand new and the setup on my Amazon instance was pristine. So when I got home I started investigating. And the only hint as to what happened were a lot of entries in /var/log/httpd/access_log, entries that looked like this:

64.137.235.207 - - [22/Jun/2016:18:19:44 +0000] "POST /xmlrpc.php HTTP/1.0" 500 251 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
64.137.235.218 - - [22/Jun/2016:18:19:45 +0000] "POST /xmlrpc.php HTTP/1.0" 500 251 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
64.137.235.218 - - [22/Jun/2016:18:19:48 +0000] "POST /xmlrpc.php HTTP/1.0" 500 251 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
64.137.235.207 - - [22/Jun/2016:18:19:50 +0000] "POST /xmlrpc.php HTTP/1.0" 500 251 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"

I had no idea what these were and what to do about them, so the first thing I did was to ask my dear friend Google. And I found this guy. I did everything he recommended and this week it happened again.

So I did what every person that manages a blog and the machine it is on. I asked the previous administrator if he knew what to do about it. And he said that these kind of attacks on my blog happened all the time when he was administering it, but he manually took the ip classes and added them to iptables with DROP. Plus, the previous server was more powerful and the effect of the attack was not visible. My micro Amazon machine is quite a defenseless victim.
So I did what could be done at this point, modify the iptables settings to be more restrictive.

#This will reject connections above 15 from one source IP.
iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 15 --connlimit-mask 32 -j REJECT --reject-with tcp-reset
#In this 160 new connections (packets really) are allowed before the limit of 150 NEW connections (packets) per second is applied.
iptables -A INPUT -m state --state RELATED,ESTABLISHED -m limit --limit 150/second --limit-burst 160 -j ACCEPT

But this is not enough, I want to shame the organizations allowing this to happen, so I created a java program to analyze the access_log file and extract ip families and organizations they match to. So below, you can see the full list of culprits:

AT&T Internet Services; Richardson
African Network Information Center; Ebene
Amazon Technologies Inc.; Seattle
Asia Pacific Network Information Centre; South Brisbane ;
CLGW; Kennett
KW Datacenter; Petersburg; 104.233.88.236
Latin American and Caribbean IP address Regional Registry; Montevideo
MCI Communications Services, Inc. d/b/a Verizon Business; Ashburn
Qwest Communications Company, LLC; Monroe
RIPE Network Coordination Centre; Amsterdam
Shaw Communications Inc.; Calgary
Time Warner Cable Internet LLC; Herndon

The requests spamming this blog came from 401 ips, I could write a script to block connections from all of them to this blog, but I don’t have the time. So I’ll just lease the rues above for iptables and I’ll see if there is need for anything else more drastic. But I really hope it won’t be for a while.
Below is the ip list, in case somebody is interested. Continue reading “xmlrpc attacks and how to handle them”


Mar 11 2016

Despre companii și ironie

Category: Miscellaneous,TechnicalIuliana @ 11:29

De ceva vreme în compania în care lucrez au loc schimbări majore. V-am povestit despre migrarea de la CVS la Git, acum că s-a cam terminat cu asta, pasul următor a fost să ne folosim de Git pentru a ne ușura viața. Zis și făcut, așa că am făcut un document în care care explicam cum se lucra pe proiecte înainte, explicam de ce nu merge acum și explicam cum ar trebui să lucram. Ar fi trebuit să fie simplu, uite ce fain, e uite ce ne ușurează munca, după următorul release facem.

Numai că lucrurile nu sunt mereu ușoare în companiile mari, pentru că sunt oameni și echipe specializate e o anumită treabă și ăia trebuie să și-o facă, nu-i așa? Indiferent dacă se pricep la ea sau nu. Și uite așa echipa de infrastructură, care se auto-numesc cu mândrie DevOps au preluat controlul GitBlit și al întregului setup făcut de mine și acum eu sunt user normal cu drept de citire și scriere doar pe un repository.

Partea ironică e că, cică eu sunt arhitect și când lucrez la câte un task, schimbarile pe care le fac sunt pe repo-uri diferite și aici începe partea distractivă, pentru că eu nu pot push-ui schimbarile mele pe repo-urile respective. Și asta pentru că un indian paranoic, care a vrut să instaleze camere de luat vederi în biroul din Sibiu să se asigure că muncim, vrea doar echipa lui, cea de infrastuctură, să aibă drepturi depline pe un tool de development. Because, fuck logic!

Și în afară de echipa de infrastructură, mai este o echipă de testare, care a modificat documentul creat de mine trasformându-l într-un monstru care urmează să fie standardul de lucru. Da, am scris bine, iar voi ați citit bine, echipa de testare, care este o chestie abstractă cu scop încă nedefinit și un lider foarte deschis la idei noi cât timp el este cel care decide implementarea. Care din punctul meu de vedere și al multora, n-ar trebui să aibă nici o treabă în a ne defini noua sistemul de development (branching) folosing Git. Mai ales când nici el și nici unul din oamenii din echipa lui n-au lucrat cu Git în viața lor.

Și mai ironic este că de când sunt arhitect mi se pare că efectiv ideile și părerile mele sunt ignorate total. În momentul de față aș prefera să dau promovarea aia înapoi și să ne întoarcem la stilul vechi de interacțiune.

Și cu adevărat ridicol este faptul că oamenii ăștia din paranoia lor și obsesia lor de a împărți oamenii pe roluri bine definite, nu realizează că efectiv se împușcă singuri în coiae. Eficiența oamenilor va scădea drastic, motivația la fel, mai ales când efectiv nu poți lucra și nu poți învăța un tool cum trebuie când poți folosi numai două funcții ale lui. E ca și cum un părinte ar vrea ca al lui copil să ajungă înnotător profesionist, dat îi dă voie să înoate doar în cadă. La un moment dat, unul din mai marii companiei mă numise a jewel, referindu-se la cunștințele mele în domeniu. Nu aș fi ajuns așa, dacă n-aș fi lucrat înainte în companii unde mi s-au permis foarte multe. Am avut drept de admin pe orice tool cu care am lucrat și de root pe mașini de producție, de îmi doream să scap de ele, pentru că era prea stresant să am atâta putere. Aici nu mai am drept de root decât pe macul meu și este uimitor cât de ineficientă sunt, pentru că efectiv când am nevoie de ceva, fie trebuie să rog pe cineva să facă asta, fie trebuie să fac un request și să aștept să îmi răspundă cineva. Deși am demonstrat că pot să mă descurc cu un drept de admin și root, dar rules are rules.

Dar m-am resemnat, dacă ei preferă să mă plătească pentru a fi ineficientă, e alegerea lor. Pe principiul cum îți așteni așa dormi, ai copiii așa cum îi crești și companiile au angajații exact așa cum și-i formează.

Stay safe, stay happy!

Tags:


Mar 10 2016

Awesome Feedback for my technical book

Category: TechnicalIuliana @ 1:34

In case you do not know I wrote a technical book, about Spring Web. From time to time, I receive emails from people reading my book and working with the code, but the email received tonight made my day:

What attracted me to your Pivotal Certified Spring Web Application Developer Exam Guide is the fact that you used Java configuration for the Spring Web Flow, and to the best of my knowledge, that is the only book in the market that currently used Java Config for Spring Web Flow. As I look further into the book I see how you encouraged the use of current and most prevalent tools for development. I love your approach, it is very upwards looking, and has the tendency of yielding a great and lasting result.
(That’s what Tim said)

So yeah, I’m a pioneer in using Java Config for Spring Web Flow. Ta da! So in case you had doubts about buying my book, I hope there are less of them now.

Stay safe, stay happy!

Tags: ,


Feb 19 2016

I’ll just leave this here

Category: TechnicalIuliana @ 10:41


Jan 06 2016

This is what I do

Category: English posts,TechnicalIuliana @ 22:25

When everybody was going on vacation me and a few other colleagues, stayed behind in order to perform the migration from CVS to Git of our very large project. We used the wonderful cvs2git tool, although a lot of internet reports say that the results are unpredictable. The same thing I mentioned during the preparatory meetings, but for the first time since I work in this company apparently there were people that were more optimistic than me, because on the 23rd of December the migration began. A little bit earlier than everybody expected, but oh well…

iuliana-rambo

We had one big CVS repository, so the first step to do was to restructure our project and split it into little ones that could be easily migrated. Issue was, that one project could not be split. And that was the one that caused a lot of trouble. When I am writing this post, that project is still being migrated. And is migrated a little different than others. Each branch of the CVS repo, becomes a Git repo. Then all these repositories will be merged into one. And all my colleagues recommended me to use this and that, a lot of shell and git commands found on stackoverflow, I had the genius spark to merge these repositories in an instant using multiple remotes. I’ll write more about this in a later post.

Before the vacation started, I trained my colleagues in using Git. If you would ask me, the training was quite a fiasco, because I had only 2 hours per group to explain them what Git is, what are the differences between CVS and Git, how Git works internally, what GitBlit is, how to work with Git using Eclipse and his stupid EGit plugin and how to work around its mishaps. As you can imagine 2 hours were not enough, but it is what it is, I had to work with the resources I was allocated. Knowing exactly how the training went, I took advantage of the free days I had and I slept a lot and prepared myself mentally for 6 months of  answering repetitive, sometimes ridiculous Git questions. I mean, I am expecting for my colleagues to have the most weird questions and I am expecting for them to do the most weird things with Git.

And now, this is the first week. And my responsibilities do not cover only Git consulting, but my project manager is on vacation so I had to take his responsibilities as my own, I had to deliver a fix and I had to prepare the hotfix package for testing and delivery and also help people in the company to update their release/hotfixes scripts to use Git. Fortunately, the hotfix was ready, was tested and will be delivered at the end of the week.

But today a serious problem emerged. People were unable to work with the remote repositories. They got a lot of timeouts, and nobody knew the cause. Logs did not say anything related to that. So we started analyzing everything it could affect this.

We started with GitBlit, all looked fine in the GitBlit.properties file, all ssh properties were set with appropriate values.

Most of us were using the ssh protocol to communicate with the remotes, so we needed to check how many ssh connections the server could handle in parallel. SSH works over TCP, so the  number of TCP connections was just as relevant.

# cat /proc/sys/net/core/somaxconn
128

And it was a small damn one. It was increased to 1024. And it seemed to work for a while, but as soon as everybody started cloning, pulling and fetching, the problem reappeared. So this was clearly not it.

I then started to look at the SSHD server installed on the server. There were two parameters that interested me: MaxSessions(specifies the maximum number of open sessions permitted per network connection) and MaxStartups(specifies the maximum number of concurrent unauthenticated connections to the SSH daemon. Additional connections will be dropped until authentication succeeds or the LoginGraceTime expires for a connection) Both were commented in our /etc/ssh/sshd_config file, so I guess the default value of 10 was used for both of them. So both were set to 1024. (Yes I like this number)

I restarted the sshd service and again for a while everything looked fine. Then the timeouts started again. I started to think that maybe GitBlit did not close the conections successfully and that is why the 1024 quota was reached and timeouts happened. So I started looking at Gitblit again. After some research into each of its properties I found this one:

# The default size of the background execution thread pool in which miscellaneous tasks are handled.
# Default is 1.
execution.defaultThreadPoolSize = 1

And you probably suspect by now… it was modified to 1024. I restarted the Tomcat hosting the GitBlit installation and… voila. Remote operations are now working for my colleagues. Apparently remote operations using the ssh protocol are miscellaneous tasks.

I was doing all these things, while consulting people about Git and my close colleagues were amazed at how serene I was and how well I was handling it all. Actually I think I was a little amazed too, but then I realized that there is nothing to be amazed of. I was prepared for this, I was expecting a hell of confusion and people running around like Dexter(the cartoon character) when his hair was on fire. I was prepared because I am good at this job and because this is what I do.

Tags: , , , ,


Dec 12 2015

What is testing?

Category: English posts,TechnicalIuliana @ 22:29

Testing is …

  • is the process used to reduce the likelihood of a failure that undertaken in a professional manner warranties that the software meets all specified user requirements  when it is delivered.
  • an exercise in risk management and reduction.
  • is a measure of quality.
  • is the process of executing a program with the intent of finding failures.
  • is a process involving static and dynamic actions in order to identify defects.
  • is the process of exercising software to detect errors and verify that it satisfies specified functional and non-functional requirements.
  • is the process of exercising software to verify that it satisfies specified requirements and to detect errors.
  • is the process consisting of all life cycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects.(ISQTB definition)
  • is the process used by testers to destroy developers sense of self esteem.
  • is the process used by testers to destroy confidence in the application.
  • is the process used by testers to help developers grow technically, develop their characters and learn to accept negative feedback.


Oct 09 2015

Just dev things

Category: TechnicalIuliana @ 16:46

After receiving an email from MyEclipse that begged me to renew my license, I got a little creative. Click on each image in order to enlarge it.
small_eclipse small-love-intellij

Julie pic source