Saturday, December 30, 2017

Impact of the compiler in XMR mining

When coming to crypto-mining the selection of the compiler can play a vital role in the performance of the application. In my box (Ubuntu 16.04 LTS - Ryzen 1600), I tried five different gcc compilers to examine the performance. Despite what someone will believe, the latest version does not offer the best performance. Moreover, the oldest compiler, that actual can compile the XMR miner has very bad performance, most likely because when it was developed the Ryzen architecture was not available. The latest version of the 4 series gcc has the best single performance, however, the benefits are non-existent when going to 8 threads. Version 5 has the best results and 6 and 7 have slightly less performance. 

I 'm providing the results of running my customized version of an xmr miner in H/s to get an idea of the performance impact after 60s execution time. I 'm really curious to learn what will happen if I compile with Intel Compiler. The Intel compiler (ICC) gave some minor performance gains in my old (!) i5 4770K CPU, but I doubt that will give any benefit.

Overall, I see a max 2-3% performance gain, except if you are using a really old compiler.


1 Core6 CoresMax (8 Threads)
gcc 4.8.547283326
gcc 4.9.476.3453504
gcc 5.4.175.8450503
gcc 6.3.075.2446492
gcc 7.2.075.2446496

Tuesday, December 26, 2017

Books for software developers professionals

As a software developer you expected to write some code. The difference between a good and mediocre developer is a combination of knowledge and skills. I compiled a list of books that are necessary to all the developers.


At the very early stage of the career, the developer should learn the basics: how to structure his code, how to write small functions, what is desirable and what is not. I can suggest two books for people starting in software development: "Code Complete: A Practical Handbook of Software Construction" from Steve McConnell and "Clean code" of Robert C. Martin. The first one presents the basic rules how to write good code, and the second one explorers in greater depths the best practices for writing good code.



 A also strongly suggest to learn about the design patterns that exists in the
software development. There are two well known books, the Design patterns : elements of reusable object-oriented software by the gang of four (Erich Gamma,‎ Richard Helm,‎ Ralph Johnson,‎ and John Vlissides) and the Head First Design Patterns by Eric Freeman,‎ Elisabeth Freeman,‎ and Kathy Sierra.









After you learned the basics of designing software, you have to move to the architecture side. There are two books that i liked most: the Clean Architecture: A Craftsman's Guide to Software Structure and Design from Robert C. Martin and the Working Effectively with Legacy Code by Michael Feathers. Both are necessary if you are involved in big projects that require both new development and maintenance of the old code. As you are getting older will also suggest to have a look to the The Mythical Man-Month: Essays on Software Engineering by  Frederick P. Brooks Jr. Old book, but provides a good view of the management of large projects.








Finally, we must not forget to train our selfs continuously. What is the best books about this practice out there? I will suggest two of them: the The Clean Coder: A Code of Conduct for Professional Programmers  from  Robert C. Martin and the The Pragmatic Programmer by Andrew Hunt and David Thomas. As a developer you never stop training and improving you self. These two books can provide the directions required for a successful career, at least as developer.



Sunday, November 26, 2017

Cryptocurrencies price

Will Bitcoin break the limit of 9000 USD? Usually prices changes in Bitcoin propagate through the other cryptocurrencies. I said few weeks ago that there are rumors about reaching 200$ USD for the Monero. If the market continues, it will reach the price, but... i strongly believe that something is broken to the system. Anyway, overall i believe the prices will increase especially for Ethereum. Time will tell...


Update: Apparently the Bitcoin was  over 9000 for some time today 26th of November 2017:


Rawtherapee Tutorial Videos

RawTherapee is one of the best cross platform Raw imaging software. It has powerful and at the same time complex tools that you can do almost anything with the pictures. The downside of course is the complexity of the application. Using only a portion of the available tools, I created few tutorial videos for starting up:


Sunday, October 29, 2017

NAS or not to NAS? - 2017 Guide

As a family we have six different external disks of various sizes ranging from 320 GBytes up to 1TB. I also have an old PC that i no longer use it and seems to be too slow even for every day tasks such as web browsing due to its mechanical disk. Then the idea of using it as Network-attached storage (NAS) come. The machine is powerful enough to provide storage inside the home.

Before building the machine i had to verify through memchek that the memory modules are in perfect condition. So, i left for 12 hours the memory check and thankfully, no error come up. Next step was to verify that each of the available disks does not have any bad sector or relocations. This can be done by using utilities to scan the disk either provided by the manufacturer (such as WD)  or use the Ultimate Boot CD.

The second step was to collect the prices of disks from local retailers. Here is a table with the model and the NAS targeted disks. I wanted something cheap, so i spitted in three configurations:


1. The cheapest choice: 2x 3TB RAID  1   + 1 TB for downloading stuff

I will select the Seagate IronWolf 3TB (119 Euro) and the WD Red 3TB (128 Euro) with a total of 247 Euro. The disk will be set in RAID 1 and the downloaded stuff in the old 1TB disk. Alternative, the 1 TB disk will be used as a backup for critical files and the 3TB disks will be also used for download. The old 320 GB will be used as a startup disk. I the future, it will be easy to upgrade using an additional 3TB disk transforming it to RAID5 and move the 1TB disk as external.

2. The wise choice: 3x 4TB RAID 5 + 1 TB for startup and  download disk.

I will select the Seagate IronWolf 4TB (170 Euro), the WD Red 4TB (190 Euro - another retailer has the price 160 but he is out of stock), and the HGST for NAS 4TB (195 Euro) with a total of 555 Euro. The primary OS disk should be replaced with a cheap SSD (40 Euro). With this configuration you can have one drive failure and 8 TB of disk space (more than enough!)

3. The expensive choice: build a system from the start using memory with error correction code. This can easy go up to 1000 Euro, as you have to buy specialized motherboard, such as Supermicro boards.

Eventually I will go with the first option, as it is the cheapest and fits very well my local needs. I will still continue using the external disk as a backup.

Saturday, October 14, 2017

A simple web scrapper

I was recently asked what is the simplest web scrapper in python that i can write. Initially i through that the Scrapy framework was the best. But then i realized it is too complex for just scrape one page that is given by console parameter. Thus, i decided to use the requests and bs4 packages.

The first step was to install them:

pip install requests bs4

After the first check of parameters using the sys.argv, i had to check if the provided URL contains the http or https prefix. The requests package expects the url to include the http or http:

if ("http" not in url):
url = "http://" + url

Next i fetch the data and get them convert them using the BeutifulSoup:
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.p
arser')


Almost all the pages today include scripts and style, so the best way to remove them is this:

# kill all script and style elements
for script in soup(["script", "style"]):
script.extract() # rip it ou
t

We usually want the body of the page and not the other parts:

extractedText = soup.find_all('body')[0].get_text()

Don't forget to transform the text to ascii to avoid any printing issues:

extractedText = extractedText.encode('ascii', errors='ignore')

Finally, it is also nice to remove the white space in the final text, especially if the input page has a lot of unicode text.
for rmtext in ('\n','\r','\t',' | ', ' , ', ' : ', ' / ',' ', ' ', ' ', ):
extractedText = extractedText.replace(rmtext, ' ')

I uploaded the script in GitHub if someone wants to play or using in any project.

Saturday, August 26, 2017

Monero (XMR) Price to 140 USD ?

Monero (XMR) currency has increased by 46%+ last hours. I almost stopped mining previous weeks, i started again. There are rumors that i will pass the 200$ mark, but i will also not surprised if the price goes down to 80$ again. It seems now that XMR is more profitable to mine compared with other cryptocurrencies. Now here is the dilemma, should i sell my XMR? It is an attractive. Anyway, if you decide to get into mining XMR, be careful!

Monday, July 31, 2017

Old town of Rhodes, Greece

Every summer I visit the old city of Rhodes in Greece. Every time i feel amazed from the liveness 
of the city. People everywhere, shops, restaurants, and of course bars. Full of night life, crowded from young and old people. Here are some impressive photos:

Rhodes, Greece, entry of the old city.
Rhodes, Greece, entry of the old city.

Sokratous street, old town, Rhodes, Greece during the night. Full of life!
Sokratous street, old town, Rhodes, Greece during the night. Full of life!

Old medieval castle of the old town in Rhodes, Greece. Impressive also in the night.
Old medieval castle of the old town in Rhodes, Greece. Impressive also in the night.

Old town of Rhodes, Greece view of the clock in one of the many small streets.
Old town of Rhodes, Greece view of the clock in one of the many small streets.


Even in the smallest streets you can find open shops during the summer. In Rhodes, Greece.
Even in the smallest streets you can find open shops during the summer. In Rhodes, Greece.


Friday, July 14, 2017

Heavy fog (again) in Nicosia

Another day with heavy fog in the morning. These photos were taken around 07:30 AM.

Heavy fog Nicosia early morning.
Heavy fog Nicosia early morning.


Heavy fog Nicosia early morning.
Heavy fog Nicosia early morning.

Heavy fog Nicosia early morning.
Heavy fog Nicosia early morning.

Saturday, June 17, 2017

Monero (XMR) mining with Ryzen 5 1600 Optimizations - How to reach 510 H/s

Recently I created a post describing the best configuration for mining XMR. After some experimentation, I transformed the famous version of the XMR-stak-cpu miner to improve slightly the performance.

The source code is available here: https://github.com/malvanos/xmr-stak-ryzen

Update May 2018: Due to new fork use the latest xmr-stak from here: https://github.com/fireice-uk/xmr-stak

After you clone the source code, you only have to write make to compile. I removed all the donation of the developers and improved 2-3% the performance in Ryzen 5 1600. To achieve the best performance with Ryzen 1600:

1. Use mild CPU overclocking: Ovecklocking from 3200 to 3700 adds only 8W in Monero mining, but gives you +50 H/s.

2. Use 8 threads: Here is the configuration for the best performance:

    "cpu_threads_conf" : 
    [ 
{ "low_power_mode" : false, "no_prefetch" : false, "affine_to_cpu" : 0 },
{ "low_power_mode" : false, "no_prefetch" : false, "affine_to_cpu" : 2 },
{ "low_power_mode" : false, "no_prefetch" : false, "affine_to_cpu" : 4 },
{ "low_power_mode" : false, "no_prefetch" : false, "affine_to_cpu" : 6 },
{ "low_power_mode" : false, "no_prefetch" : false, "affine_to_cpu" : 8 },
{ "low_power_mode" : false, "no_prefetch" : false, "affine_to_cpu" : 10 },
{ "low_power_mode" : false, "no_prefetch" : false, "affine_to_cpu" : 12 },
{ "low_power_mode" : false, "no_prefetch" : false, "affine_to_cpu" : 14 },

    ],

I have already modified the file in the repository. Note that the threads with affinity 12 and 14 actually jump all over the place, but the final out come is quite nice.

3. Use huge pages: I include the huge.sh script. You have to run as root. The command to allocate large pages is sudo sysctl -w vm.nr_hugepages=256

4. Run as root to enable the memory locking.

Some minor optimizations are underway, I will try to schedule them in the future. Here is the result:

Ryzen 1600 XMR Mining
Ryzen 1600 OC 3.7GHz - 510 H/s XMR

Sunday, May 14, 2017

Monero (XMR) mining with Ryzen 5 1600 and RX 460


Update June 2017: reached 510 H/S with a different configuration file.

As I discussed earlier, I recently bought a new machine for my home. After assembled the machine and booted up, I discovered I had to make a  BIOS update to be able to run with DDR4 in 2400 MHz. I also tried to increase the frequency to 2600, but no luck. At least I was able to decrease the latency of the memory. Anyway, after some tuning, I was able to overclock the CPU to 3700 without any voltage increase and without any serious increase in power consumption. The Ryzen comes with a nice cooler that allows basic overclocking without any need for additional cooling, although the summer is here, and maybe if few days I will have to take care it.

Anyway, saw in some forums that the people were having bad performance with XMR (Monero) mining and I decided to give a try. I was able with mild overclocking (to 3700) to achieve 450-480 H/s in contrast to their reports of 360 H/s.  You have manually to download the source and compile the files. So here is my instructions/tips to achieve better performance :

1. Use linux: I noticed better support and scheduling of processes. The performance of miners in windows was really bad!

2. Update the BIOS: This will allow you to decrease the DRAM latency and increase the DDR4 frequency. I noticed a significant increase from 380 H/s to 410 H/s.

3. Use large pages: In Linux is easy, just write sudo sysctl -w vm.nr_hugepages=256.

4. Use the proper miners: I 'm using the https://github.com/OhGodAPet/cpuminer-multi for the CPU and https://github.com/fireice-uk/xmr-stak-amd for the GPU. Note: switched to XMR-stak-ryzen (500 H/s), more information here
Update May 2018: Due to new fork use the latest xmr-stak from here: https://github.com/fireice-uk/xmr-stak

5. Use mild CPU overclocking: Ovecklocking from 3200 to 3700 adds only 8W in Monero mining, but gives you +50 H/s.

6. Use 8 threads: Although Ryzen 5 1600 has 6 physical cores after the BIOS update using 8 threads seems to benefit the overall performance.

7. Remove donation by editing the source files: In most open source miners you can modify the files responsible. For example, the xmr-stak you have to edit the donate-level.hpp file.

To compile efficiency you have to edit the CPU miner and make the following changes:

1. Remove preferch instruction from the cryptonight_aesni.c: Inside the function cryptonight_hash_ctx, the prefetch instruction limits the unroll. This will allow increasing the performance by a small margin.

2. Compile with gcc-6 and -march=znver1: To achieve the best performance you have to use the latest version of the compiler that supports microarchitecture optimizations.

I will cover the GPU optimization in Linux in a future post. Until then, mine...




Due to high cost of RX 4XX/5XX cards i would suggest to search also Ebay for possible cheaper solutions for RX 560. It is not worth it paying more than 160$.

Tuesday, May 2, 2017

Ryzen 5 1600 first thoughts

Spend one day to install and modify my new PC. I selected a Ryzen 5 1600 with 16 GB of RAM with an ASRock AB350 PRO4. I also installed a 850 SSD to speedup the IO. First impressions: really fast. The Linux part (Ubuntu 16.04) was mature enough, but the windows was not working out of the box. The only downside was that i had to update the BIOS of the motherboard to 2.5 to make the DDR4 to work in 2400 Mhz. The second issue was that i had to use the AMD pro drivers for my RX 460 to use the full potential of OpenCL.



Played a bit with XMR (monero) mining. Despite what people are saying for bad performance in mining, i discovered 400-430 H/s with 7 threads and huge pages.





Sunday, April 23, 2017

Temperature variation in Cyprus


Many are thinking that the weather in Cyprus is always nice and hot. The reality is that it varies through the year.

This is picture taken 3 months ago in Troodos (end of February), on of the mountains of Cyprus. Everything was covered with snow and the temperature was around 0. Also for small periods of time the temperature inside the capital city of Cyprus can go -1 or -2.

Troodos snow Cyprus.
Troodos snow Cyprus.

And then, the spring came. We got a heat wave from Africa and the result was 32*C during the day inside the city of Nicosia. This picture was taken mid April. As we entering to the summer, we expect even higher temperature, up to 40*C.

Temperature Nicosia April
Temperature near Nicosia.

Summer here we come! Can't wait the time to visit the beautiful beaches again!
A beach near Larnaka, Cyprus.
A beach near Larnaka. Notice the clean waters, the picture was take without any polarized filter.


Saturday, February 25, 2017

Recovering data from external USB driver

Almost two weeks ago, I tried to connect one of my external USB hard drive to my desktop, but without any success. The files I wanted most were already backed up to a different disk, so I was not expecting a bit loss. However, I had a set of benchmarks that could be useful in the future. Ok I said, it is not a high priority, but how can I recover my data?

Here is a number of steps I followed trying to recover the disk:

1. The disk is a WD passport USB drive. I noticed when I was moving the USB cable, the driver will disconnect from the PC. So the first obvious step was to try a new USB cable, but no luck. There was a small improvement in the transfer speed, but nothing serious (more on that later).

2. So the second thought was to open the case and connect directly the disk to the desktop PC. Unfortunately, the disk has embedded the USB controller and it cannot be attached directly to the PC with  SATA cable.

WD Passport USB disk controller with USB connector.
WD Passport disk controller with USB connector.
A bit of more Google search revealed that newest drivers also encrypt the data, even if the user didn't set a password.

3. Then I tried to put the disk in the refrigerator, but it didn't do anything. A bit of warning here, this could help if the disc is stopped somewhere in the plate. Some people warned that this can actual destroy your disk. If your data are not critical or you are desperate, you can try to see what happens.

4. The next step is to try mounting the disk as read-only under Linux and use software recovery tools. This requires more experience than the average Windows user, but you can try using a live CD without installing anything. It is important to understand that the Linux must be able to see the device, even if you can't see the actual files.

In Linux, the storage devices and partitions are stored under the /dev/sd*. For example, in my system the default devices are:

[email protected]:~# ls /dev/sd*
/dev/sda  /dev/sda1  /dev/sda2  /dev/sda3  /dev/sda4  /dev/sda5 /dev/sda6 /dev/sdb /dev/sdb1

The /dev/sda is the primary disk, and the /dev/sdb is the external USB connected device. Not that I have dual boot machine, so I have different partitions for Windows. /dev/sdb is the external USB disk and the /dev/sdb1 is the partition.

If you can't see the partition, then you can't mount the disk (see step 6). If you can't see the disk at all, you are out of luck. You can try switching the controller of the disk using the same model, but it will not work in the case of WD passport disks due to encryption. An alternative when using external USB disks is to connect the USB cables directly to the controller, but you need some experience with electronics. I'm going to analyze this case in the future article.

All the new Linux distributions automatically mount the disk in read-write configuration. I really suggest that even if Linux is able to mount and see the files, to unmount the disk and remount it as read-only. To do so, you need to find where is mounted and force umount. For example, as root:

[email protected]:~# mount | grep sdb
/dev/sdb1 on /media/wd type fuseblk (rw,nosuid,nodev,allow_other,default_permissions,blksize=4096)

[email protected]:~# umount --force /media/wd
[email protected]:~# mkdir  /media/wd-recover/
[email protected]:~# mount  -o ro /dev/sdb1 /media/wd-recover/

After the mount, you should be able to see the files, even in low speed.

5. If you can mount the disk, the next step is to try recover files from the disk. There are two famous tools that you have to use: ddrescue and PhotoRec (includes the TestDisk). From the time you can mount the disk, you should use the ddrescue to copy files that resist copying (for our example file.iso):

Create a directory. If you are using live CD try to connect an external disk and make the recovery directory there:

mkdir recover
cd recover

Recover the data without trying again. This is a first step to speed up the speed.

ddrescue --no-split -v -c 256 -A /media/wd-recover/file.iso file.iso logfile

Now let it retry previous errors 3 times, using uncached reads. This can be really slow:

ddrescue -v -d -r 3 /media/wd-recover/file.iso file.iso logfile

Now let it retry previous errors 5 times, using uncached reads:

ddrescue -v -d -r 5 -c 1 /media/wd-recover/file.iso file.iso logfile

Now let it retry previous errors 5 times, using uncached reads in reverse:

ddrescue -v -d -r 5 -c 1 -R /media/wd-recover/file.iso file.iso logfile

In any time, you can stop the process and continue in the future. The progress and the status are saved in the logfile.

6. If you can's see the partition, then you have to create an image of the disk and then use the photorec tool. You follow the same process as step 5 with ddrescoue, but in this case, you copy the entire disk. The downside is that you must have enough space. In addition, disks with 1TB or more are taking a long time for coping the data. For example:

Recover the data without trying again. This is a first step to speed up the speed.

ddrescue --no-split -v -c 256 -r 1 -A /dev/sdb image.dd logfile

Now let it retry previous errors 3 times, using uncached reads. This can be really slow:

ddrescue -v -d -r 3  /dev/sdb image.dd logfile

Next, you must use the testdisk to restore the partitions or recover deleted files:

testdisk image.dd

If you are able to restore the partition tables (check with fdisk -l image.dd), you can mount the file directly to your machine and read the files:

For example if the output of fdisk is:

Units = sectors of 1 * 512 = 512 bytes

Device Boot      Start         End      Blocks   Id  System
image.dd   *         128     8015999     4007936    b  W95 FAT32

You have to write (512*128 = 65536): 

mount -o loop,offset=65536 image.dd /mnt/tmp

7. If you can't restore the partitions, then you have to use directly the photorec to the image file.

photorec image.dd 

You follow the instructions and recover the data (will be analyzed in the future).

Friday, January 20, 2017

RTL-SDR Blog V3 Arrived

I recently bought a new RTL dongle that supports Direct Sampling that allows frequencies less than 40Mhz. In particular, the model I ordered was the RTL-SDR BLOG V.3. DONGLE that was really cheap compared with other solution. It still can't reach the quality of the other more expensive receivers, but it still a step up. The Dongle comes with a long external antenna.

RTL-SDR Blog V.3 Dongle
The RTL dongle.

So, the question now is how better is from my old SDR. I did a check with the RTL power tool to see what is the difference. My old SDR Dongle (Fitipower FC0013) has coverage from 22 to 1100 MHz. The new Dongle RTL Blog V3 a has Rafael Micro R820T has coverage from 24 - 1766 MHz, but it also contains Direct Sampling that allows for High Frequencies. A word of warning here, the reception using Direct Sampling is very bad, especially if you connect the antenna without a filter band or/and preamplifier. I did experiments using the rtl_power, and the results showed much more gain for the new Dongle. All the experiments were done using the internal antenna inside the home. I don't have yet the ferrites installed (post is really slow here), but you can get an idea how the dongles are doing. Here are the plots (click to zoom):

Fitipower FC0013 24-500 MHz Band survey.
Fitipower FC0013 24-500 MHz Band survey.

RTL Blog Rafael Micro R820T 24-500 MHz Band survey.
RTL Blog Rafael Micro R820T 24-500 MHz Band survey.

However, we should be interested in the Signal Noise Ratio rather to actual power received. The new Dongle is more sensitive, but it also gets more noise. I'm waiting for the ferrite to arrived to get a clear view the noise of the new Dongle. Here is a screenshot receiving HF:

Receiving HF with a lot of noise.
You can buy the receivers here: SDR Blog Receiver with Rtl2832u ADC Chip.