https://www.siddharthjain.dev/posts/Recent content in Blog on Siddharth JainHugo -- gohugo.ioWed, 27 Oct 2021 21:31:30 -0700https://www.siddharthjain.dev/posts/2021/zanzibar-consistent-auth-system/Wed, 27 Oct 2021 21:31:30 -0700https://www.siddharthjain.dev/posts/2021/zanzibar-consistent-auth-system/Zanzibar is an authorization system developed at Google and used by more than 1500 client services, including Calendar, Cloud, Drive, Maps, Photos, and Youtubehttps://www.siddharthjain.dev/posts/2021/foundationdb-transactional-key-value-store/Tue, 12 Oct 2021 20:45:09 -0700https://www.siddharthjain.dev/posts/2021/foundationdb-transactional-key-value-store/FoundationDB is an open source key-value store and one of the first systems to combine the flexibility and scalability of NoSQL architectures with the power of ACID transactions.https://www.siddharthjain.dev/posts/2020/designing-data-intensive-applications-sstables-lsm-trees-b-trees/Sat, 24 Oct 2020 19:29:58 -0700https://www.siddharthjain.dev/posts/2020/designing-data-intensive-applications-sstables-lsm-trees-b-trees/An LSM-Tree is at the heart of storage engines that are based on the principle of merging and compacting sorted files. Such storage engines are typically called LSM storage engines.https://www.siddharthjain.dev/posts/2020/designing-data-intensive-applications-hash-indexes/Sat, 17 Oct 2020 11:32:24 -0700https://www.siddharthjain.dev/posts/2020/designing-data-intensive-applications-hash-indexes/Many databases internally use a <em>log</em> which is an append-only sequence of records. These are <em>log-structured</em> databases. Hash indexes are used to index key-value data, and may be used to power log-structured databases.https://www.siddharthjain.dev/posts/2020/designing-data-intensive-applications-reliability-scalability-maintainability/Fri, 02 Oct 2020 19:18:33 -0700https://www.siddharthjain.dev/posts/2020/designing-data-intensive-applications-reliability-scalability-maintainability/A data-intensive application is one for which raw CPU power is rarely a limiting factor — it has bigger concerns over the amount of data, complexity of data and the speed at which the data is changing.https://www.siddharthjain.dev/posts/2020/information-retrieval-part-1-boolean-retrieval/Sun, 05 Jul 2020 22:48:33 -0700https://www.siddharthjain.dev/posts/2020/information-retrieval-part-1-boolean-retrieval/Information Retrieval (IR) is the science of searching for material in a collection of resources based on an information need. The most common form of an IR system is a text retrieval system that allows users to retrieve textual data in the form of a sorted list of documents using natural language queries.https://www.siddharthjain.dev/posts/2020/zookeeper-wait-free-coordination-for-internet-scale-systems/Sun, 10 May 2020 21:49:20 -0700https://www.siddharthjain.dev/posts/2020/zookeeper-wait-free-coordination-for-internet-scale-systems/ZooKeeper is a service that allows distributed processes to coordinate with each other using a shared name space of data registers. It exposes a wait-free interface and an event-driven mechanism to provide a simple and high-performance kernel for building distributed applications. Originally developed at <em>Yahoo!</em>, it is now under the care of the <em>Apache Software Foundation</em>.https://www.siddharthjain.dev/posts/2020/epidemic-algorithms-for-replicated-database-maintenance/Thu, 23 Apr 2020 22:29:39 -0700https://www.siddharthjain.dev/posts/2020/epidemic-algorithms-for-replicated-database-maintenance/This post is a summary of the paper: Epidemic algorithms for replicated database maintenance, Demers, et al., 1988 Introduction Published in the late 80&rsquo;s, this paper lays out early ideas on gossip based replication algorithms and focuses on eventual consistency, in contrast to the traditional ACID model. The idea of eventual consistency or BASE is well-known these days, however, in the late 80&rsquo;s, it was one of the most important novel ideas.https://www.siddharthjain.dev/posts/2020/web-scale-responsive-visual-search-at-bing/Sun, 23 Feb 2020 23:01:45 -0700https://www.siddharthjain.dev/posts/2020/web-scale-responsive-visual-search-at-bing/This post is a summary of the paper: Web-Scale Responsive Visual Search at Bing, Hu et al., 2018 Introduction Visual search is an interesting research area. A visual search system ranks a list of visually similar images when presented with a query image. As in all search systems, latency and relevance of returned results are key metrics to evaluate such a system.https://www.siddharthjain.dev/posts/2020/unravelling-constexpr/Tue, 28 Jan 2020 20:24:04 -0700https://www.siddharthjain.dev/posts/2020/unravelling-constexpr/Background C++ is an old language. The excitement that C++ generated in the 90&rsquo;s had faded by 2010, probably because of the emergence of several newer languages like Java and also because the Standards Committee had released only a few enhancements in the decade. And then came C++11. The specification for C++11 was approved in 2011 (hence, the name).https://www.siddharthjain.dev/posts/2019/peloton-resource-scheduling-at-uber/Sun, 15 Dec 2019 20:30:32 -0700https://www.siddharthjain.dev/posts/2019/peloton-resource-scheduling-at-uber/Peloton is Uber&rsquo;s cluster scheduler that is capable of co-scheduling mixed types of workloads such as batch, stateless and stateful jobs in a single cluster for better resource utilization. Designed for scaling to millions of containers and tens of thousands of nodes, it features advanced resource management capabilities like elastic resource sharing, hierarchical max-min fairness, workload preemption to name a few.https://www.siddharthjain.dev/posts/2019/anatomy-of-large-scale-hypertextual-web-search-engine/Wed, 06 Feb 2019 22:43:09 -0700https://www.siddharthjain.dev/posts/2019/anatomy-of-large-scale-hypertextual-web-search-engine/This is a well-known paper. It was published in 1998 and it describes Google as a prototype of a large-scale search engine. According to the authors, this paper was the first in-depth public description of a large-scale web search engine. The authors describe how to build a practical large-scale search system that exploits information present in hypertext, thus producing better search results than existing search systems.