draskolnikova.github.io

KADABRA - IDNIC Smart Routing Analyzer

Sat, 05 Dec 2020 00:00:00 +0700

It’s been 2 years ago I’ve updated my blog :( But thanks for anyone who’s still reading this dead blog (lol). This post will tell about my latest project in last 2-3 year that involved many people and I hope useful to other people too.

Today, I’d like to write about my latest projects with IDNIC called KADABRA. KADABRA is smart routing analyzer platform and helps IDNIC or network administrator to analyze internet routing table, especially in Indonesia.

KADABRA v1 built at 2018 in collaboration with RIPE Atlas, Bukalapak and IDNIC-APJII as sponsor. You can read the paper about first KADABRA release. It’s talk about transparency and simple network mitigation using traceroute and ping only. And we faced so many problems because of RIPE Atlas framework limitation and we didn’t have any control access into RIPE backends.

The first development takes 1 year, since end of 2018 until end of 2019 and the result wasn’t as expected due the probes limitation.

KADABRA v2

In early 2020, we tried to rebuild KADABRA from scratch and collecting several issue that we facing in KADABRA v1, especially at IDNIC. So, we decided to build KADABRA as a Smart Routing Analyzer to help IDNIC validates the data and helps the Indonesian Network Administrator debuging network problem in the internet.

KADABRA v2, born at Feb 2020 with several improvement and adjustment. KADABRA have several core values:

Collecting and analyze all bgp attributes both ipv4 and ipv6.
Marking RPKI with ROA Status (Valid, Invalid and Unknown for each IP Blocks)
RIS (Routing information service) realtime stream with 5 minutes update interval Indonesian domestic routing information base
It’s track the growth of new IPv4, IPv6 and new ASN in daily estimate.
… and much more!

KADABRA built on top heterogen ecosystem that makes the application and infrastructure more modular and decentralized.

Infrastructure

All infrastructure platform distributed across multi-cloud (agnostics), with the following details :

JETDINO Hosting - Kubernetes Platforms
Alibaba Cloud - Relational Database Management System
Biznet GIO - Network Object Storage
Cloudflare - Anycast Domain Name system
Google Cloud Platform - Cloud-based message queueing

And all infrastructure above are hosted in Indonesia.

Application

KADABRA Application built using open sources framework, with the following details :

Go Programming Language (Backend)
Laravel Framework (Frontend)
PostgreSQL Database (Database)

Statistics

Currently, KADABRA already record approx. 96K prefixes across 5 exchange in 4 cities. And we have processing power up to 33K request per second to analyze the prefix and create public statistics via RAGNO.

People behind KADABRA is :

Ardika Bagus as Senior Cloud & Kubernetes Engineer
Indra Saputra as Senior Backend Engineer
Fahri Ahmad Fadil as Application Engineer
Lufti Rahadian as Senior System & Network Engineer
Abdul Basit as Senior Database Architect
Achmad Reyhan as Network Engineer Support
Adi Nugroho as System Engineer Support

Special thanks to:

Chief of IDNIC - Mr. Adi Kusuma
Chief of APJII - Mr. Jamalul Izza
Chief of PT. APIK Media Inovasi - Mr. Azhari Ahmad
Indah Yuliani - KADABRA Logo & Design

… and all people behind this project, especially it’s you! Yes you!

If you have any question or feature request regarding KADABRA, you can send an email to abra [at] kadabra [dot] id or you can directly mention me at twitter @draskolnikova

Set transparent RTBH to upstream using MikroTik RouterOS

Sun, 01 Apr 2018 00:00:00 +0700

Couple weeks ago, there was memcrashed exist in the wild and it’s hurt my router so hard. It was attack my router up to 1GBps, fortunately my router doesn’t hung. LoL.

How to mitigate this issue from client side?

There’s several options to resolve this issue, you can drop it using firewall or send your ddos’ed IP Address to blackhole. To save your router(s) resource, it’s simply send your ddos’ed IP to upstream blackhole.

Then what happen if you’re a service provider and using MikroTik RouterOS? You need to pass your internal bgp communities to upstream bgp communities.

How to passing internal blackhole communities to upstream blackhole communities?

For example, my client has ddos’ed IP 192.168.10.10, my blackhole community is [AS:6969], my upstream blackhole community is [UPSTREAM:999] and client ASN is AS65666.

TL;DR

(AS65666) => (AS65555) => (AS65444)

AS65666 = Client ASN AS65555 = Service Provider AS65444 = Upstream Provider

From client point of view (AS65666), they need to blackhole and mark IP 192.168.10.10 disappear from internet, so they need send blackhole community to (AS65555) and pass it to (AS65444)

From AS65666 point of view:

/rou filter
chain=transit-out-AS65555 prefix=192.168.10.10 prefix-length=32 invert-match=no action=accept set-bgp-prepend-path="" \
     set-bgp-communities=65555:6969 append-bgp-communities=""
     
/rou bgp network
add network=192.168.10.10/32 synchronize=no

AS65666 Advertisement

[client@as65666] > /routing bgp advertisements print peer-as65555
PEER         PREFIX               NEXTHOP          AS-PATH    ORIGIN     LOCAL-PREF
peer-as65555 192.168.10.10/32     172.16.66.2                 igp
peer-as65555 192.168.10.0/24      172.16.66.2                 igp

From AS65555 point of view:

/rou filter
add action=accept bgp-communities=65555:6969 chain=transit-out-65444 comment=RTBH set-bgp-communities=65444:9999
add action=accept bgp-communities=65555:6969 chain=transit-in-65666 prefix-length=29-32

Prefix length in AS65555 is how long prefix we accept for blackhole community.

AS65555 Advertisement

[dewangga@as65555] > /routing bgp advertisements print peer-as65444
PEER         PREFIX               NEXTHOP          AS-PATH    ORIGIN     LOCAL-PREF
peer-as65444 192.168.10.10/32     172.16.20.2      AS65666    igp
peer-as65444 192.168.10.0/24      172.16.20.2      AS65666    igp

We can see advertised prefix above, we found 192.168.10.10 should be blackholed and sent to AS65444 as upstream provider. It’s just a simple case, and you can improve using sflow or any opensource tools.

Creating unlimited storage using Google Cloud Storage

Mon, 28 Aug 2017 00:00:00 +0700

Couple months ago, I have simple task but hardly to deploy. Creating huge/big storage for rawset/dataset but using very effience budget. Sounds impossible? Yes, it is. Because, to deploy redundant storage server it requires “huge” on everything (money, infrastructure, etc). The requirements is so simple, user stored assets, save it and access it.

So, I decided to put it on the cloud. I choose Google Cloud Storage (GCS) instead of AWS S3. Why? I need FUSE, and only GCS have official FUSE support, called gcsfuse.

By default, GCS will not allow you to create any files and directory then sync them to the cloud. But, they gave many parameters to allow you do that (like local file system on your linux).

First Init

First thing first, don’t forget to install and auth your self on Google by follow the instruction. If you’re successfully authorized, you should get information like this:

[jenna@galea ~]$ gcloud config list
[compute]
region = asia-east1
zone = asia-east1-a
[core]
account = [removed]
disable_usage_reporting = False
project = machine-learning-[removed]]

Your active configuration is: [default]

Mounting buckets

I assume you already have buckets, if don’t please follow the instruction(s). The buckets called buckets1.

Please select or create your key by following this instructions. If you have create and download the key, the key should be like this.

{
  "type": "service_account",
  "project_id": "machine-learning-[removed]",
  "private_key_id": "f3b9d87f5837200[removed]",
  "private_key": "YOUR_PRIVATE_KEY_HERE",
  "client_email": "[removed]-compute@developer.gserviceaccount.com",
  "client_id": "113014412[removed]",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/[removed]-compute%40developer.gserviceaccount.com"
}

Then you can place it anywhere, don’t forget to chmod 600 otherwise the mount progress won’t work. Then on /etc/fstab, put your configuration like this.

buckets1 /mnt/buckets1   gcsfuse rw,allow_other,file_mode=0644,dir_mode=0755,implicit_dirs,key_file=/root/key-files.json

Trying mount it using mount -a command, if you got no error, you should access the directory. Try to put some files/directory on this directory, it will appears also on the cloud and vice versa.

Hope it helps! :)

Creating bridge interface using nmcli

Wed, 23 Aug 2017 00:00:00 +0700

Recenly I got confused setting up VLAN and deliver it to my virtualization server(s). I pretty sure my VLAN configuration was working fine and double check the logic is working. I have 10 (Ten) KVM Host, 9 nine of them was properly connected to networks (bridged interface(s)) and the other is having problem because of the bridge interface won’t up.

Problem

The interface can’t connect to existing networks, it can be caused by manual bridge interface creation.

Since the network configuration is a bit differents between EL6 and EL7, nmcli is the key here. Trying to hook up manual configuration by adding brX interface on /etc/sysconfig/networks-script, but no luck.

Have you setup your bridge interface using this old configuration ?

TYPE=Bridge
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=br0
DEVICE=br0
ONBOOT=yes
IPADDR=ip.ad.dr.es
PREFIX=24
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes

If yes, trying to avoid that, and you should (must!) using nmcli.

To create bridge interface, you just only type this in your servers.

[root@linux ~]# nmcli con add type bridge ifname br0 stp yes priority 36864
[root@linux ~]# nmcli con add type ethernet ifname enp1s0f0 master bridge-br0

Where enp1s0f0 is the interface and change with your real interface name, then br0 is new bridge interface, you can named it, and it’s not always br0 (can be br1, br2, etc). And if you want to see script files inside /etc/sysconfig/networks-script/, it should be like this :

DEVICE=br0
STP=no
TYPE=Bridge
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=bridge-br0
UUID=411a4a13-caab-41e1-83f5-4d1c31ab0fb4
ONBOOT=yes
IPADDR=192.168.69.25
PREFIX=24
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes

Conclusion

If you’re using NetworkManager on EL7, you should avoid creating or editing network configuration located under /etc/sysconfig/networks-script. Use NetworkManager (available on nmcli or nmtui) instead of manual.

Fixing VLC flickering on AMD Graphic Cards

Wed, 09 Aug 2017 00:00:00 +0700

Couple days ago, I’ve been update my Fedora Core 25 to latest Fedora Core 26. Got some adjustment and improvement there, also got new (bugs?) errors regarding my dedicated graphic cards. From lspci reports, I am using Radeon HD 8670A/8670M/8690M / R5 M330 / M430 graphic cards on Lenovo G40-80 notebook.

$ sudo lspci -v | grep AMD
04:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Sun XT [Radeon HD 8670A/8670M/8690M / R5 M330 / M430] (rev 83)

When I launch VLC, the video was flickering like this.

And I search for the clue, why this VLC flicker when I start it using dedicated graphic cards, I found it there.

Trying to check default VLC Configuration, it was set to automatic.

Then, I was trying set it to VA-API video decoder via DRM.

Don’t forget to save, apply and restart your VLC.

Test again my latest configuration, and voila!

I hope it helps if you got same problem like me.

How to deploy and migrating to IPv6

Mon, 07 Aug 2017 00:00:00 +0700

Since I (accidentally) work as Network Administrator, I always curios with IPv6 and how to deliver it to customer (especially my own device). But please keep in mind, you need up-to-date firmware and network device (router, switch, wireless radio(s)) and another equipment to do this.

In my lab and production environment, RouterOS firmware and MikroTik device(s) are still capable to do this transition. Why? It’s cheap, easy to configure and “rock-solid”. So, my setup actually like this :

1x MikroTik RB850Gx2 (tested on RoS 6.39.2)
1x HP v1920-24G (Managed)
4x HP v1410-2G (Unmanaged)
1x TP-Link TL-SF1048 (Unmanaged)
2x UAP AC Pro

As you can see, looks like this :

The goals is to reduce my routing policy because of NAT (with simple routing, I can easily reach client device than using NAT). It’s can be solved using IPv4 too, but we already know, how IPv4 were exhausted.

How to deploy it?

Configuration on Router MikroTik RB850Gx2

Assume that you already have /64 prefix from your ISP to do Router Advertisement (RA).

For example, I use this prefix 2400:ff:ec00:dead::/64 and ether3 is interface point to HP v1920. (please review and change the configuration mathced on your side)

[admin@MikroTik] > /ipv6 address add address=2400:ff:ec00:dead:: advertise=no interface=ether3
[admin@MikroTik] > /ipv6 nd
set [ find default=yes ] disabled=yes
add advertise-mac-address=no interface=ether3 managed-address-configuration=yes mtu=1500 other-configuration=yes reachable-time=10s \
    retransmit-interval=5s
/ipv6 nd prefix
add interface=ether3 prefix=2001:df2:cc00:dead::/64
/ipv6 nd prefix default
set autonomous=no

Interesting point there, if you are using RouterOS too and have /127 point-to-point links from your upstream or ISP, then your ISP(s) using MikroTik too, you need to add the default route using local-links.

/ipv6 route
add !bgp-as-path !bgp-atomic-aggregate !bgp-communities !bgp-local-pref !bgp-med !bgp-origin !bgp-prepend check-gateway=ping distance=1 gateway=\
    fe80::e68d:8cff:fe3f:6731%ether5 !route-tag

Don’t know why, but it works.

Since now, you can check in your /ipv6 neigh, you should get advertised prefix from your client. Cheers and welcome to IPv6!

Taking network monitoring to the next level

Sat, 12 Nov 2016 00:00:00 +0700

As system engineer and part-time network engineer, I am responsible to make sure all system are running and the network properly connected (24/7/365).Now, I will write about a project came from RIPE NCC, called RIPE Atlas. RIPE Atlas is a global network of probes that measure Internet connectivity and reachability, providing an unprecedented understanding of the state of the Internet in real time.

Currently, I host 2 probes (the device from RIPE is called probe), 1 public probe #25259 and 1 private probe #25577. My public probe have dual stack network configuration, both IPv4 and IPv6 are enabled. You can create custom measurement, for example I have infrastructure(s) on Amsterdam, and have FQDN speedtest.ams01.softlayer.com, then I create public measurement to monitoring my network route.

From that measurement, I can track down the history of my network route, whenever it was down or reroute to another link. It’s help me so much to maintain the SLA from upstream. The probe have another capability to monitoring, available feature is PING, TRACEROUTE, DNS, SSL, HTTP, and NTP. And you can select specific probe(s) to fulfill your monitoring needs.

This project is FREE*, you only need to host the probe online. The atlas probe includes: 1xTP-Link MR3020 (Modified F/W), 1xUTP Cable, 1xMicroUSB Cable, 1x8GB USB Disk. Interest to join the project? Feel free to reach me out on twitter, or you can directly contact to Atlas Probe Ambasaddor at Indonesia using Twitter or Facebook (Mr. Budiwijaya).

Happy probing!

Building AKAMAI-like CDN using Amazon AWS with minimal budget

Sat, 29 Oct 2016 00:00:00 +0700

Based on Gartner Magic Quadrant (Aug) 2016, AWS still leader on cloud IaaS provider. And I’ve been using Amazon AWS since 2013 (S3 Bucket, Cloudfront, and Route53). The reason why I use that service only on Amazon AWS are, to optimize existing on premise infrastructure. The big challenge is to manage big assets (images) without downtime and data loss and of course using optimized resource.

According to S3 SLA, I believe that at least I have only 0.01% downtime, it means only approximately 43minutes in a month. And what happen if the S3 bucket can’t accessed in period time? I combine with Amazon CloudFront. CloudFront will cached anything that hosted by AmazonS3.

Actually, the design is looks like.

I use S3 bucket to store all assets as origin, then distribute them through CloudFront. And I use Route53 as my DNS Servers, since route53 support geolocation and failover support, it will be nice. Cloudwatch? It’s help me to monitor performance of cache server on Indonesia, if the cache server goes down or unreachable, it will be trigger route53 to remove it from dns record and switch to AWS CloudFront distribution as default images server (static.xtremenitro.org).

Cost Calculation

It’s the most important to do setup using Amazon AWS, miss configuration, you’ll be charged more expensive. From the existing installation, I have.

Amazon Simple Storage Service (S3)

I thought it’s still the best storage service ever, because I have a lot of asset(s) there, approx 400GB++, it’s only charged me $14.xx.

Amazon CloudFront

Since CloudFront didn’t have any node in Indonesia, build cache server in Indonesia (our visitor is 99% in Indonesia) will cut off your Amazon CloudFront billing usage.

Amazon Route53

It’s the key to save your resource, using Amazon Route53, and the feature of Geolocation and Failover combine with CloudWatch.

VPS on Indonesia

The last one, to serve our visitor came from Indonesia, we need at least two VPS with 2 vCPU & 2 GB of RAM. Why we need this? We need lower latency to serve Indonesian Visitor. You can go through JetdinoVM using Professional package (this is non-paid article, seriously).

Total Cost

Amazon S3: $14.61
Amazon CloudFront: $43.80
Amazon Route53: $44.87
VPS: $60

Total Cost: $163.28 (round up $165)

What if we use all of AWS Infrastructure? I’ve tried it couple months ago, it will charged you $450 / month. And still having some latency issue regarding internet access in Indonesia.

With this algorithm and specification, I can reduce the monthly cost and increase speed access to user upto 1-3 times.

What about akamai? The title as I mentioned above. Since now, only Akamai that penetrate CDN in Indonesia, most CDN company nearby nodes is on Singapore. And what about the price? I have the price, but it’s NDA, you can directly contact Akamai’s Sales for more information.

Subnetting IPv6 Cheatsheet

Thu, 27 Oct 2016 00:00:00 +0700

Since IPv6 hype was launch several years ago, many people (incl. me) were struggling how to split IPv6 subnet correctly. Honestly, I wasn’t network engineer, but learning network fundamental is not a big deal (especially IPv6, nowadays, it’s a new protocol!). Since it was launch at 1990 (former IPNG), based on APNIC Labs Survey, the adoption of native IPv6 still low.

The fundamental of TCP/IP is how to correctly split the subnet and know the routing works. At July 25th, I was attend the Advance IPv6 Routing Workshop by APNIC and learn with many senior engineers right there. The most interesting part is, I know how to split IPv6 subnet correctly without using any IPv6 calculator. I’ll write down how to do it for self notes, and I hope it is usefull for you too.

Subnetting

Prefix IPv6: 2404:6800::/32 (Google IPv6 Block Asia Pacific)

We must know that IPv6 have 128-bit address, separated by colon (:) and have 8 (eight) groups, then IPv4 only have 32-bit address, separated by dots (.) and only have 4 (four) groups.

So, if the rest IPv6 have 128-bit address, then each group should have 16-bit.

Shortest IPv6 Prefix: 2404:6800::/32 128-bit addresses

Longest IPv6 Prefix: 2404:6800:0000:0000:0000:0000:0000:0000 128-bit addresses

Then if I want split it into small subnets (eg. /48), how? And which first prefix should be?

For the example, we will split IPv6 Prefix 2404:6800::/32 into /48, then found the 48-bit.

2404:6800:0000:0000:0000:0000:0000:0000  -> 128-bit total
 16   16   16   16   16   16   16   16   -> 128-bit total (from 16-bit x 8 block)

Please keep in mind, IPv6 is a hexadecimal. So it will start counting from 0 to f

Then, in every member in a group(s) have 4-bit address [0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f]

How we found that 48-bit address from 2404:6800::/32 ?

So we must find the 48th bit from 2404:6800::/32, then we need 16-bit left.

Diagram

  [4]
0 0 0 0 -> each number represent 1-bit operation(s)
| | | |
| | | +---- 0 1 2 3 etc (2^0)
| | +------ 0 2 4 6 etc (2^1)
| +-------- 0 4 8 c etc (2^2)
+---------- 0 8 0 8 etc (2^3)

2404 on first group, have first 16-bit address.

6800 on second group, have second 16-bit address, so total bit are 32-bit right now.

0000 on third group, have third 16-bit address, so total bit are 48-bit right now.

Voila! We’ve found the 48th bit. The 48th bit are on third group, and first IPv6 is 2404:6800:0::/48 on /48 prefix.

Then another case, how we found /33 prefix? You can use the referrence diagram above and let’s find out.

IPv6 on /32 Subnet: 2404:6800:0000:0000:0000:0000:0000:0000

IPv6 on /33 Subnet: 2404:6800:0000:0000:0000:0000:0000:0000

See the marked zero 0 on /33 subnet, it’s the hint to find the subnet, and crosscheck with the diagram. So the /33 prefix should be 2404:6800:0::/33 and 2404:6800:8::/33.

Hope it helps!

Creating optimized qcow2 files

Wed, 12 Oct 2016 00:00:00 +0700

Standard and default images create using virt-manager on linux, automatically full allocating specified disk size to host. To avoid un-usable disk space, you should avoid choosing default images created by virt-manager.

-><-

To create growing images files for VM’s, I choose qcow2. Why growing images? To ensure the host only allocated how much diskspace to harddrive (fill-as-you-go), not how big the disk was allocated. For example :

$ sudo qemu-img create -f qcow2 phantom01.xtremenitro.org-lvm-f.qcow2 100G
Formatting 'phantom01.xtremenitro.org-lvm-f.qcow2', fmt=qcow2 size=107374182400 encryption=off cluster_size=65536 lazy_refcounts=off

It should be create qcow2 files and have following information:

$ sudo qemu-img info phantom01.xtremenitro.org-lvm-f.qcow2
image: phantom01.xtremenitro.org-lvm-f.qcow2
file format: qcow2
virtual size: 100G (107374182400 bytes)
disk size: 196K
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false

See at disk size, we have only 196K on first init. For sure, let’s check it.

$ ls -lah phantom01.xtremenitro.org-lvm-f.qcow2
-rw-r--r--. 1 dmnQ dmnQ 194K Oct 13 03:35 phantom01.xtremenitro.org-lvm-f.qcow2

And it will save your disk space as much as you want, and will growth as the data growth. Then after creating the disk file, you need to load it on virt-manager by choosing Select or create custom storage.

-><-

Done, now you have optimized qcow2 files and growth as the data growth.