Re-balancing Deflate Compression Levels
klauspost 0 31st January 2016 Go, Programming, WebAfter I have spent a lot of time optimizing the Go deflate compression, I found that a lot of my benchmarks were looking like this:
asmfmt: Go Assembler Formatter
klauspost 0 16th December 2015 Assembler, Go, ProgrammingOnce you get used to working with gofmt on a daily basis, you want to have it everywhere. I have been writing a good amount of assembler for Go, and I was getting annoyed by not having similar functionality for assembler.
Travis/AppVeyor CI Script for Go
klauspost 0 4th November 2015 Go, Programming, testingContinous Integration tests are an important part of maintaining package stability. Since Travis has recently made it possible to run tests on OSX and with AppVeyor having Windows support, we now can create free CI tests for all the major desktop platforms.
Fast Stream Deduplication in Go
klauspost 13 22nd October 2015 Go, ProgrammingThis post will introduce the concept of deduplication and a Go package I have written to help you implement it. It is inspired by the work on the zpaq journaling archiver done by Matt Mahoney. If you are looking for a pure commandline tool, I can also recommend SuperRep by Bulat Ziganshin. This will however provide you…
An Async Read-ahead Package for Go
klauspost 1 11th October 2015 Go, ProgrammingAfter looking at some rare poor performance in rclone, I have created a package for read-ahead functionality, which I have implemented a few times in different contexts. This gave me the opportunity to create a solid implementation with solid tests to make sure it works with good performance. For the impatient here is the package:…
The case of the subtle race condition
klauspost 3 29th September 2015 Go, ProgrammingYesterday I got a strange race condition reported after I wrote some additional tests for pgzip. At first I couldn’t really work out why I was getting the race reported, and I suspected that the race detector was reporting wrong line numbers. The race reported was boiled down like this:
Creating a “Load Balancing Reverse Proxy” in 1 Work Week
klauspost 0 25th September 2015 Go, Programming, WebLast week I gave myself the challenge to create something in a week. Not a mockup, but a complete usable solution with documentation and enough tests to make it feel solid. I have been looking at DigitalOcean for some time, and while browsing their documentation I came across a sample script that would help you set…
Long Windows paths (UNC paths) in Go
klauspost 0 17th September 2015 Portability, ProgrammingLately I have been working a bit with various backup/file tools written in Go, mainly rclone and restic. I noticed that both of them has issues when dealing with long Windows paths, which is quite unfortunate for backup utilities, so I figured that might be a good thing to cover on my blog. This issue…
[Re:Post] Interpolated look-up tables using SSE2
klauspost 0 9th September 2015 Assembler, Image Processing, Programming, RePostUpdate 2015: This is a re-post of a 2010 article I wrote for the Rawstudio blog, which at the time of writing is off-line.
Intrinsics in Go
klauspost 0 4th September 2015 Assembler, Go, ProgrammingI have written a Discussion Piece about how it could be possible to implement assembler intrinsics in Go based on an experiment I have been working on the past week. The goal of the experiment was to see if intrinsics are feasible within the Go language without breaking the language. It is my impression that…
Fast Linear Time Gzip/Zip Compression
klauspost 1 26th August 2015 Go, Programming, WebThis is the final chapter in our trilogy of exploring deflate performance. In this I will give a detailed description of how you can get close to constant time compression for gzip/zip and standard deflate running at approximately 150MB/s per core.
How to make your Go Applications easy to port
klauspost 0 18th August 2015 Go, ProgrammingGo is really well suited for cross-platform work. My main development environment is Windows, but I always deploy to Linux-based systems. Therefore I naturally try to avoid things that will get me into trouble. My attitude towards cross-platform support is that if you consider yourself a serious developer, your code should at least compile on…
Gzip Performance for Go Webservers
klauspost 13 5th August 2015 Go, Programming, WebIn our previous performance test, we looked at high throughput compression, suited for log files, map-reduce data, etc. This post will deal with webserver performance, where we will investigate which gzip is best suited for a Go webserver. Update (18 Sept 2015): Level 1 compression has been updated to a faster version, so the difference…
Go gzip/deflate benchmarks
klauspost 5 31st July 2015 Go, Programming, WebAs a continuation of my release of a Optimized gzip/zip packages. I have done some informal benchmarks to get an impression of performance characteristics of the various libraries. I have compiled the results in a spreadsheet you can view yourself. Update (18 Sept 2015): Level 1 compression has been updated to a faster version, so the difference to…
Optimized gzip/zip packages, 30-50% faster
klauspost 5 28th July 2015 Assembler, Go, ProgrammingI have worked on optimizing the standard library deflate function, and I am happy to announce revised gzip/zip packages, that on x64 is about 30-50% faster with slightly improved compression.
defer fun()
klauspost 0 25th July 2015 Go, ProgrammingAfter programming Go for more than two years my favorite Go keyword is defer. I might go into detail about this in a future post, but this is one of the nice things I found. About a year ago I came across a funny little Go-ism when I was looking through some code. It was…
Adding Unsupported Instructions in Golang Assembler
klauspost 0 9th July 2015 Assembler, Go, ProgrammingFor an experimental project, I wanted to use an SSE 4.1 instruction (MPSADBW) for some Go assembler. I found out that this instruction wasn’t supported by the Go assembler in the current 1.4.2 version. The proposed solution is to add this as raw opcodes in the assembler. The instruction (in Go assembler syntax) is: MPSADBW…
Big Password Dictionary for Validation
klauspost 0 8th July 2015 Security, WebContinuing my previous posts [1][2] on checking passwords against dictionaries, I have completed a dictionary for the extremely paranoid. It is a compilation of the ‘Crackstation‘, ‘g0tmi1k‘ and ‘WPA-PSK WORDLIST 3 Final’ dictionaries. The passwords are all in lower-case, Unicode NFKD normalized, unique and sorted. I also dropped all passwords shorter than 8 characters and more…
Protect Your Passwords Against Dictionary Attacks
klauspost 0 6th July 2015 Go, ProgrammingAs a followup to my previous blog post: “Password Requirements Done Better“, I have finalized a package for dictionary password validation for Go. If you have a site where users have passwords, and you don’t already do dictionary checks, I would strongly recommend you look at this. The package can be used to protect your…
Password Requirements Done Better
klauspost 0 30th June 2015 Programming, Security, UX, WebWe have all encountered them: Password Requirements. They are a well-intended attempt to make sure that your personal information stays hidden, and only you can access your websites. However, there are still many very annoying password requirements, and attempts at visualizing a “password strength” to help you choose a good password. However, I find that…
Blazingly fast Reed-Solomon Coding
klauspost 5 22nd June 2015 Go, ProgrammingLast week Backblaze published a blog post on their usage of Reed-Solomon Erasure coding to maintain data available even if some of their storage servers goes offline. They also released a java library with the code they use for creating the reed-solomon sets. I have always been fascinated by Reed-Solomon Erasure coding, but so far I…
XAudio2 Dolby Digital Live patcher
klauspost 2 7th June 2015 UncategorisedFor many years a bug has remained in XAudio2 library, that will make XAudio 2 report stereo channels to games using the API, even though output is 5.1. This will make sound cards with DDL encoding output stereo instead of nice surround-sound. I found that this was still an issue, even with a fully updated…