Since first premiering in 2018, Bluey has been a favorite between kids and parents alike (it’s no surprise the show has a 9.4 rating on IMDB). For five years, our littles have been growing up with The Heeler family. Over the summer, Disney+ dropped 10 ...

5 parenting lessons we learned from ‘Bluey’

There’s a horrific war going on between Israel and Gaza. Many of us have been glued to the nonstop news coverage and social media feed as the horror unfolds before our eyes. The question for all ...

Israel blockade of Gaza is inhumane despite ghastly attack by Hamas

Simply sign up to the Climate change myFT Digest -- delivered directly to your inbox. Last week, the term “energy transition” was mentioned more than 3,000 times in news reports, media ...

We are in a period of climate stuckness

About 1.1 million people living in northern areas of Gaza have been told to leave ahead of an expected ground offensive by Israel. In this video, two people from across the territory share their ...

Voices from across Gaza: We don't know where we can go

We are increasingly using AI to create and maintain relationships with customers, but the real short-term need is in creating and maintaining relationships with our employees. IBM is a client of t ...

Why we need to focus AI on relationships

As Israel continued to press its assault on Gaza after Hamas infiltrated by land, sea and air over the weekend, killing and capturing over 1,000 Israelis, the spotlight has focused on the militant ...

What is Hamas? What to know about the group that launched attacks on Israel

Much is unclear about the strike that Palestinian authorities say killed hundreds. By The New York Times Hundreds of people were feared dead after an explosion at a hospital in the Gaza Strip on ...

What We Know About the Explosion at the Hospital in Gaza

Cambuslang’s Kenny McLean says Scotland’s new-found belief has taken them to the Euro 2024 finals in Germany. Steve Clarke’s side qualified with two games to spare when Spain beat Norway 1-0 ...

Scotland's Euro 2024 qualification happened because we have more belief, says star

Until quite recently there was a near consensus among forecasters that the U.S. economy was headed for a recession. In fact, it’s been exactly one year since Bloomberg declared that, according ...

Soft Landing, Here We Come?

A group of mums spent just £102 on an 18-hour spa trip to Milan – and were home in time for the school run. Laura Oduntan, 37, and her pals - Jade Bassett, 33, Remi Pedro, 37, and Agnes ...

'We're mums and flew to Milan for an 18-hour holiday - and got back in time for school run'

WeGroup

wegroup

Authors:
(1) Yufan Zhang, Duke Kunshan University, SciEcon CIC (yz605@duke.edu);
(2) Zichao Chen, Duke Kunshan University, SciEcon CIC (zc142@duke.edu);
(3) Luyao Zhang, Duke Kunshan University. SciEcon CIC (lz183@duke.edu);
(4) Xin Tong, Duke Kunshan University (xin.tong@dukekunshan.edu.cn). Authors: Authors: (1) Yufan Zhang, Duke Kunshan University, SciEcon CIC (yz605@duke.edu); (2) Zichao Chen, Duke Kunshan University, SciEcon CIC (zc142@duke.edu); (3) Luyao Zhang, Duke Kunshan University. SciEcon CIC (lz183@duke.edu); (4) Xin Tong, Duke Kunshan University (xin.tong@dukekunshan.edu.cn). CONTENT OVERVIEW Abstract and 1 Introduction 2 Methodology 3 Results 4 Discussions and Conclusion, and References ABSTRACT As a blockchain-based application, Non-Fungible Token (NFT) has received worldwide attention over the past few years. Digital artwork is the main form of NFT that can be stored on different blockchains. Although the NFT market is rapidly developing, we observed potential ethical and racial fairness issues in the design of NFT artworks due to a lack of ethical guidelines or censorship. Therefore, we investigated CryptoPunks, the most famous collection in the NFT market, to explore and visualize its potential ethical issues. We explored the ethical issues from three aspects: design, trading transactions, and related topics on Twitter. We scraped data from Twitter and Dune Analytics using python libraries, Twitter crawler, and sentiment analysis tools. Our five visualizations implied that 1.6 times more male punks were created in the initial design process than the female ones. And the male ones have a higher average selling price than females; lighter-skinned punks tend to sell for higher prices. The results of our study and visualizations provide a preliminary exploration of CryptoPunks and further inspire future ethical-related investigation and research in the NFT domain. 1 INTRODUCTION Non-fungible token (NFT) is a unit of data stored on the blockchain that certifies a digital asset to be unique [14]. The NFT market achieved more than 2 billion dollars in transactions in the first quarter of 2021 — which is 2000% higher than the fourth quarter of 2020 [10]. Besides, many famous artists and companies have joined the NFT market [9, 13]. Numerous NFT digital avatar collections are created based on attributes like the avatar’s gender, skin color, and accessories. CryptoPunks [1] is one of the most popular NFT collections that offers more than 10,000 unique collectible character images, selling on every mainstream NFT marketplace. However, potential ethical problems arise in the fledgling NFT market without a regulatory mechanism and central government. For example, CryptoPunks with different genders and skin colors have significantly different price trends [8]. Moreover, the algorithmic or manual generation processes of NFT usually remain unrevealed. Although CryptoPunks creators claimed that each artwork is generated algorithmically, the punks showed imbalance attributes in gender and race [1]. Therefore, our research questions of this project are: • What potential gender and racial ethical concerns exist in NFT artwork collections in the generation and trading process, such as CryptoPunks? • What are Twitter users’ perceptions of NFT ethical problems? 2 METHODOLOGY 2.1 Data Source The data are from two data sources: blockchain transaction records of CryptoPunks token from DuneAnalytics [2] ranging from the launch date of CryptoPunks, June 23, 2017, to May 7, 2022, and the Twitter threads with at least five likes from Twitter [15] with related topic keywords including “NFT,” “CryptoPunk(s),” “ethic,” “informed consent,” “transparency,” “accountability,” “privacy,” “fairness”, “trust,” “gender,” “ethnicity,” “skin tone,” and “skin color.” 2.2 Data Processing 2.2.1 Construct databases Using the Python Pandas [12] and Numpy [11] libraries, we construct four databases: transaction database (16,823 transaction records), token database (10,000 punks), trader database (5,911 addresses), and tweet database (83,568 tweets), whose primary keys are the transaction ID, CryptoPunks ID, Ethereum address, Twitter thread ID respectively. 2.2.2 Extract visualization data To depict the distribution of CryptoPunks with different attributes, we group the CryptoPunks tokens with their attributes. Specifically, the attributes of CryptoPunks are categorized into four levels: the type (e.g., human), the gender (e.g., male), the skin tone (e.g., dark), and the number of attributes (e.g., 3 attributes). Furthermore, the transaction database is merged with the token database to add the CryptoPunks ethics-related attributes (i.e., gender and skin tone) into each transaction record. Using the Python NetworkX [4] library, we build the CryptoPunks transaction network for each year from 2017 to 2022 with the addresses as the nodes and the transaction as the edges. 2.2.3 Conduct sentiment analysis Using the Python flairNLP [7] library, we conducted sentiment analysis on the tweet content in the tweet database. Unlike the rule-based sentiment analysis toolkit, such as Textblob [6], flairNLP processes the texts via the deep neural network, which generally yields better model performance in Natural Language Processing (NLP) tasks such as sentiment analysis. Specifically, all the tweets are classified into positive, neutral, or negative sentiments. 2.3 Data Visualizations We construct five visualizations using the Python Plotly library [3]. First, we create an interactive Sankey diagram to illustrate the distribution of CryptoPunks with different attributes. Second, we apply the Time River design idiom to visualize the price change of different combinations of attributes. Third, we create a scatter plot to depict the price disparity of skin tone. Fourth, we create a circular network visualization based on the undirected graph of the transaction network by year. Last, we create the bar chart and word cloud figures using Python WordCloud [5] package to visualize Twitter users’ sentiment towards NFT-related ethical topics. A detailed introduction to the visualization results will be shown in the results section. 3 RESULTS 3.1 CryptoPunks Generating Distribution Knowing the initial amount of CryptoPunks provides insights of the design team’s opinions of equality. Figure 1 reveals intuitively that the number of genders in the original set of CryptoPunks is quite different. There are 6,039 male Punks, but only 3,840 female Punks. There were also differences in the number of punks with different skin colors, with dark skin having the lowest number. 3.2 Price Trend of Punks in Different Skin Colors and Genders In this section, we analyzed the price changes of eight types of Punks in the market and the differences between them, as shown in Figure 2 (from top to bottom are ”Male Albino”, ”Male Light”, ”Male Medium”, ”Male Dark”, ”Female Albino”, ”Female Light”, ”Female Medium”, ”Female Dark”). Figure 2 indicates that the average transaction prices of male and light-skinned punks was higher than that of female and dark-skinned punks of the same period at most time from year 2017 to now. However, the price gaps among these groups were not very huge. 3.3 Transaction Prices Sorted by Skin Color Type The scatter plot visualizes the CryptoPunks transaction records and its skin tone. Each circle represents a transaction, and its color refers to the punk’s skin tone. Through interaction, users can see the CryptoPunks details and data for each transaction. In general, the dark spots were generally below the light ones, which implied that the price of dark Punk is generally lower than that of light Punk. 3.4 CryptoPunks Transaction Network (by Year) The circular network visualizes the transaction network of CryptoPunks by year, which explores CryptoPunks’ current trading market and trading network. Each node refers to a trader who has participated in at least one transaction of CryptoPunks, and each edge refers to a transaction with the color representing the punk skin tone of the transaction. In this graph, the right side of the transaction line is the most dense, which means that prices in the CryptoPunks market are likely to be dominated by a few big business merchants or high frequency collectors. 3.5 Twitter Analysis with Bar Chart & Word Cloud The bar chart and the word cloud visualizations in Figure 5 reflected the sentiments of Twitter users’ attitudes about NFT ethics. We did not identify many participants who are aware of gender and ethical inequities in NFT, which means that the ethical training and management of the NFT market is urgently needed and necessary. 4 DISCUSSIONS AND CONCLUSION NFT is a scorching and popular market where the numbers of players are growing rapidly. In our project, we successfully collected and visualized data about the market transactions of CryptoPunks, an NFT artwork collection and tweets. Our five interactive visualizations demonstrated that different levels of gender and racial inequality have emerged both during the creation and trading processes of CryptoPunks. Our project, as a pioneer in this field, will resonate with the community and more players and inspire more researchers to study the ethical issues of blockchain. REFERENCES [1] Cryptopunks: All attributes. [2] Dune. [3] Modern analytic apps for the enterprise - plotly. [4] Networkx — networkx documentation. [5] Wordcloud for python documentation — wordcloud 1.8.1 documentation. [6] Textblob: Simplified text processing — textblob 0.15.2 documentation, 2018. [7] flairnlp/flair, 09 2020. [8] M. Egkolfopoulou and A. Gardner. Even in the metaverse, not all identities are created equal. Bloomberg.com, 12 2021. [9] R. Farrington. Why big brands are spending millions on nfts. [10] R. Frank. Nft sales top $2 billion in first quarter, with twice as many buyers as sellers, 04 2021. [11] Numpy. Numpy — numpy, 2009. [12] Pandas. Python data analysis library — pandas: Python data analysis library, 2018. [13] M. Swant. As coca-cola auctions its first nft, more brands are entering the metaverse. [14] V. Tabora. Looking beyond nft from million dollar jpeg energy hogs to its value proposition, 04 2021. [15] Twitter. Twitter. it’s what’s happening., 2000. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

CryptoPunks: The Digital Art Boom with an Analog Bias Problem

Authors:
(1) Ben Athiwaratkun, AWS AI Labs;
(2) Sujan Kumar Gonugondla, AWS AI Labs;
(3) Sanjay Krishna Gouda, AWS AI Labs;
(4) Haifeng Qian, AWS AI Labs;
(5) Sanjay Krishna Gouda, AWS AI Labs;
(6) Hantian Ding, AWS AI Labs;
(7) Qing Sun, AWS AI Labs;
(8) Jun Wang, AWS AI Labs;
(9) Jiacheng Guo, AWS AI Labs;
(10 Liangfu Chen, AWS AI Labs;
(11) Parminder Bhatia, GE HealthCare (work done at AWS);
(12) Ramesh Nallapati, Amazon AGI (work done at AWS);
(13) Sudipta Sengupta, AWS AI Labs;
(14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction 2. Related Work 3. Background 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 6. Conclusion and References A. FAQs B. Related Work C. Setup D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques C. Setup C.1. Model Training Details We trained multiple models with varying sizes, ranging from 125 million parameters to 13 billion parameters, using code data with a context size of 2048 and adjusting the per-GPU batch size and total number of steps according to the model size. For model training we used multiple p4 instances each equipped with 8 40GB Nvidia A100 GPUs per instance. For our largest model family, the 13 billion parameter model, we used a global batch size of 1024, which approximately translates to 2 million tokens per batch. The settings for each model within each model-size family were kept consistent. The remaining training hyperparameters are summarized in the following table 1. C.2. Model Configurations For each model size we train three models with attention variations; multi head where g = h, multi group where 1 < g < h and multi query where g = 1. Additionally, for 672m and 2.8b models we train a multi group model variant where the fanout in feed forward layer is decreased from 4×d to 2×d. Each model variant yields different number of total parameters therefore we group these models into family of model sizes. The detailed architectural choices for each of the model family is found in the table 2. C.3. Model Details of 1B Latency Experiment In Section 5.2.2, we use candidate models of sizes roughly 1B to study the effect of bifurcated attention. We outline the hyperparameters of such models below. C.4. Ablation Studies: 2d Intermediate Feature Dimension One can also argue that different g results in different balance of the number of parameters in the feedforward versus the attention components. We performed an ablation study where we reduce the typical intermediate feature size of 4d to 2d and train models for three model sizes (which we will refer to as the 2d experiment). The ablation study reveals that the scaling laws curves for the 2d experiment crosses the usual 4d curves, which implies that the reduced size of the attention component alone compared to feedforward does not provide a consistent explanation of model capabilities. This can be seen from Figure 9. C.5. Inference Setup We use Nvidia A100 GPUs for inference hardware (Choquette et al., 2021). We perform latency studies using Deepspeed inference (Rasley et al., 2020) on top of Huggingface transformers (Wolf et al., 2019), where we wrote custom code to handle the generalize multi-group attention as well as bifurcated attention. Future work includes extending the implementation to FasterTransformer (NVIDIA). This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Authors: Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Work 2. Related Work 3. Background 3. Background 3.1. Notation and 3.2. Language Model Inference 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 5.3. Applications 6. Conclusion and References 6. Conclusion and References A. FAQs A. FAQs B. Related Work B. Related Work C. Setup C. Setup D. Multi-Group Attention Family D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention E. Context-Aware Bifurcated Attention F. Applications: Additional Results F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques G. Compatibility with Speculative Decoding and Fast Decoding techniques C. Setup C.1. Model Training Details We trained multiple models with varying sizes, ranging from 125 million parameters to 13 billion parameters, using code data with a context size of 2048 and adjusting the per-GPU batch size and total number of steps according to the model size. For model training we used multiple p4 instances each equipped with 8 40GB Nvidia A100 GPUs per instance. For our largest model family, the 13 billion parameter model, we used a global batch size of 1024, which approximately translates to 2 million tokens per batch. The settings for each model within each model-size family were kept consistent. The remaining training hyperparameters are summarized in the following table 1. C.2. Model Configurations For each model size we train three models with attention variations; multi head where g = h, multi group where 1 < g < h and multi query where g = 1. Additionally, for 672m and 2.8b models we train a multi group model variant where the fanout in feed forward layer is decreased from 4×d to 2×d. Each model variant yields different number of total parameters therefore we group these models into family of model sizes. The detailed architectural choices for each of the model family is found in the table 2. C.3. Model Details of 1B Latency Experiment In Section 5.2.2, we use candidate models of sizes roughly 1B to study the effect of bifurcated attention. We outline the hyperparameters of such models below. C.4. Ablation Studies: 2d Intermediate Feature Dimension One can also argue that different g results in different balance of the number of parameters in the feedforward versus the attention components. We performed an ablation study where we reduce the typical intermediate feature size of 4d to 2d and train models for three model sizes (which we will refer to as the 2d experiment). The ablation study reveals that the scaling laws curves for the 2d experiment crosses the usual 4d curves, which implies that the reduced size of the attention component alone compared to feedforward does not provide a consistent explanation of model capabilities. This can be seen from Figure 9. C.5. Inference Setup We use Nvidia A100 GPUs for inference hardware (Choquette et al., 2021). We perform latency studies using Deepspeed inference (Rasley et al., 2020) on top of Huggingface transformers (Wolf et al., 2019), where we wrote custom code to handle the generalize multi-group attention as well as bifurcated attention. Future work includes extending the implementation to FasterTransformer (NVIDIA). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Training AI Models on Nvidia A100 GPUs

Authors:
(1) Howard Zhong, MIT CSAIL (howardzh@mit.edu);
(2) Mark Hamilton, MIT CSAIL, Microsoft (markth@mit.edu). Table of Links Abstract and 1 Introduction 2 Methodology 3 Results 3.1 CryptoPunks 3.2 Aggregate NFT Market 4 Discussion 5. Conclusion/ Acknowledgements/ References A Appendix A.1 Implementation Details A.2 Detailed NFT Information & A.3 Google NFT Searches Map A.1 Implementation Details We elaborate on the methods and the implementation details. This sections follows the structure Section 2. A.1.1 Initial Data Collection Our dataset only includes NFTs transacted on OpenSea, which is the primary marketplace for NFTs on the Ethereum blockchain. We query the OpenSea “v1/collections” endpoint at the end of November 2022 [15] to retrieve collection metadata as well as each individual NFT’s metadata and last sale price. We choose 790 collection names from the Kaggle Ethereum NFTs dataset [11] and NFTs from the top 30-day and all-time OpenSea volume leaderboard around November 2021. Most collections have around 5,000 to 10,000 items, and our queries yield around 70-80% of the data as the other 20-30% do not have transactions. After querying the Application Programming Interface (API) using a script written with the PySpark framework [17] on Databricks, we create a dataframe containing each NFT’s collection name, id, sale information, metadata, and image url. We then loop through this dataframe with multiple workers to download the images in parallel. In the end, we obtain a dataset of about ∼ 2.5 NFTs, each that have been transacted upon. A.1.2 Retrieving Race and Gender Labels To get gender-labelled data, we select collections that have metadata with the words “male” and “female” to find collections where at least 50% of the images have gender labels. We filter out collections that have more than 75% male or female to avoid rarity factors affecting the price. We find a total of 44 such collections with gender labels that satisfy the criteria. These collections with gender labels are It is more challenging to determine the race of an NFT collection because most collections do not directly annotate this information. For the CryptoPunks collection, we label each NFT based on the frequencies of the four different pixel values representing the four different skin tones as displayed in Figure 6. The above process was inspired by [5]. For our analysis, we group darkest and mid as Dark and lighter and lightest as Light. For other collections, we were unable to retrieve race labels from the metadata. We also attempted to find race labels for a subset of the 44 collections with gender labels with the criteria that there were a high proportion of items that could be categorized as either dark or light skinned. We filtered out collections that have more than 75% light-skinned or dark-skinned to avoid rarity factors affecting the price. This required manual review because most collections do not have easy-to-search race labels in the metadata, or even way to categorize race. For Avastar, we categorized “Amber Brown” as dark-skinned and “Pale Pink” as light-skinned in the “skin tone” attribute. For Dynamic Duelers, we categorized “Black” as dark-skinned and “White” as lightskinned in the “Skin color” attribute. A.1.3 Statistical Tools to Analyze Bias in Gender and Race We detail the statistical methods used to analyze the bias in NFT pricing for gender and race. Specifically, we go into more detail about the implementation and underlying assumptions of these tests. Unpaired t-test assumes independence of observations and each point belongs to either male or female


the variances of male and female prices are equal for the population


the prices follows a normal distribution


no significant outliers in each group The unpaired t-test is robust to all but large deviation from these assumptions. Assumption 1 and 2 are satisfied, but 3 and 4 require more careful consideration. Log Transformation: For assumption 3, with rare NFTs worth significantly more than common NFTs, NFT price distributions tend to follow a power law distribution [14]. Because it’s well known that stock prices follow a log-normal distribution [1], we apply the same transformation and found log of price tended to be closer to a normal distribution. We thus decided to also run the t-test on the log of prices, which we refer to as log t-test. Outlier Trimming: For assumption 4, as outliers may occur due to very high selling prices for rare NFTs or very low selling prices due to humans errors while listing, we address this issue by Winsorization [19], or trimming outliers past a certain percentile. Specifically, we report t-test results while trimming both left and right tails of the distribution at 0.1%, 1%, 2.5%, and 5% percentiles. Paired t-test: In addition to the unpaired t-test for independent samples, we also run a paired t-test for two dependent samples in order to better isolate the male versus female price difference while fixing time. Specifically, we calculate t-statistics on the paired difference of male and female prices marked to the daily mean price and the weekly mean price of the NFT. Assumptions 1 and 2 are now (1) male - female price must be continuous and (2) observations are independent of one another. Assumptions 3 and 4 from unpaired test remain the same and. We address assumptions 3 and 4 with the log-transformation and Winsorization, as described earlier. The approach described above is also used to compare the prices of lighter-skinned and darkerskinned CryptoPunks. We utilize the SciPy Python package [21] to implement both types of t-tests. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Howard Zhong, MIT CSAIL (howardzh@mit.edu); (2) Mark Hamilton, MIT CSAIL, Microsoft (markth@mit.edu). Authors: Authors: (1) Howard Zhong, MIT CSAIL (howardzh@mit.edu); (2) Mark Hamilton, MIT CSAIL, Microsoft (markth@mit.edu). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Methodology 2 Methodology 3 Results 3.1 CryptoPunks 3.1 CryptoPunks 3.2 Aggregate NFT Market 3.2 Aggregate NFT Market 4 Discussion 4 Discussion 5. Conclusion/ Acknowledgements/ References Conclusion/ Acknowledgements/ References A Appendix A Appendix A.1 Implementation Details A.1 Implementation Details A.2 Detailed NFT Information & A.3 Google NFT Searches Map A.2 Detailed NFT Information & A.3 Google NFT Searches Map A.1 Implementation Details We elaborate on the methods and the implementation details. This sections follows the structure Section 2. A.1.1 Initial Data Collection Our dataset only includes NFTs transacted on OpenSea, which is the primary marketplace for NFTs on the Ethereum blockchain. We query the OpenSea “v1/collections” endpoint at the end of November 2022 [15] to retrieve collection metadata as well as each individual NFT’s metadata and last sale price. We choose 790 collection names from the Kaggle Ethereum NFTs dataset [11] and NFTs from the top 30-day and all-time OpenSea volume leaderboard around November 2021. Most collections have around 5,000 to 10,000 items, and our queries yield around 70-80% of the data as the other 20-30% do not have transactions. After querying the Application Programming Interface (API) using a script written with the PySpark framework [17] on Databricks, we create a dataframe containing each NFT’s collection name, id, sale information, metadata, and image url. We then loop through this dataframe with multiple workers to download the images in parallel. In the end, we obtain a dataset of about ∼ 2.5 NFTs, each that have been transacted upon. A.1.2 Retrieving Race and Gender Labels To get gender-labelled data, we select collections that have metadata with the words “male” and “female” to find collections where at least 50% of the images have gender labels. We filter out collections that have more than 75% male or female to avoid rarity factors affecting the price. We find a total of 44 such collections with gender labels that satisfy the criteria. These collections with gender labels are It is more challenging to determine the race of an NFT collection because most collections do not directly annotate this information. For the CryptoPunks collection, we label each NFT based on the frequencies of the four different pixel values representing the four different skin tones as displayed in Figure 6. The above process was inspired by [5]. For our analysis, we group darkest and mid as Dark and lighter and lightest as Light. For other collections, we were unable to retrieve race labels from the metadata. We also attempted to find race labels for a subset of the 44 collections with gender labels with the criteria that there were a high proportion of items that could be categorized as either dark or light skinned. We filtered out collections that have more than 75% light-skinned or dark-skinned to avoid rarity factors affecting the price. This required manual review because most collections do not have easy-to-search race labels in the metadata, or even way to categorize race. For Avastar, we categorized “Amber Brown” as dark-skinned and “Pale Pink” as light-skinned in the “skin tone” attribute. For Dynamic Duelers, we categorized “Black” as dark-skinned and “White” as lightskinned in the “Skin color” attribute. A.1.3 Statistical Tools to Analyze Bias in Gender and Race We detail the statistical methods used to analyze the bias in NFT pricing for gender and race. Specifically, we go into more detail about the implementation and underlying assumptions of these tests. Unpaired t-test assumes Unpaired t-test independence of observations and each point belongs to either male or female the variances of male and female prices are equal for the population the prices follows a normal distribution no significant outliers in each group independence of observations and each point belongs to either male or female independence of observations and each point belongs to either male or female the variances of male and female prices are equal for the population the variances of male and female prices are equal for the population the prices follows a normal distribution the prices follows a normal distribution no significant outliers in each group no significant outliers in each group The unpaired t-test is robust to all but large deviation from these assumptions. Assumption 1 and 2 are satisfied, but 3 and 4 require more careful consideration. Log Transformation: For assumption 3, with rare NFTs worth significantly more than common NFTs, NFT price distributions tend to follow a power law distribution [14]. Because it’s well known that stock prices follow a log-normal distribution [1], we apply the same transformation and found log of price tended to be closer to a normal distribution. We thus decided to also run the t-test on the log of prices, which we refer to as log t-test. Log Transformation: Outlier Trimming: For assumption 4, as outliers may occur due to very high selling prices for rare NFTs or very low selling prices due to humans errors while listing, we address this issue by Winsorization [19], or trimming outliers past a certain percentile. Specifically, we report t-test results while trimming both left and right tails of the distribution at 0.1%, 1%, 2.5%, and 5% percentiles. Outlier Trimming: Paired t-test: In addition to the unpaired t-test for independent samples, we also run a paired t-test for two dependent samples in order to better isolate the male versus female price difference while fixing time. Specifically, we calculate t-statistics on the paired difference of male and female prices marked to the daily mean price and the weekly mean price of the NFT. Assumptions 1 and 2 are now (1) male - female price must be continuous and (2) observations are independent of one another. Assumptions 3 and 4 from unpaired test remain the same and. We address assumptions 3 and 4 with the log-transformation and Winsorization, as described earlier. Paired t-test: The approach described above is also used to compare the prices of lighter-skinned and darkerskinned CryptoPunks. We utilize the SciPy Python package [21] to implement both types of t-tests. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Behind the Scenes: Data Collection and Statistical Tools for NFT Bias Study

Authors:
(1) Diwen Xue, University of Michigan;
(2) Reethika Ramesh, University of Michigan;
(3) Arham Jain, University of Michigan;
(4) Arham Jain, Merit Network, Inc.;
(5) J. Alex Halderman, University of Michigan;
(6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad;
(7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction 2 Background & Related Work 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 10 Discussion and Mitigations 11 Conclusion 12 Acknowledgement and References Appendix 6.2 ACK-based Fingerprinting OpenVPN engages in a TLS-style handshake with its peer over the control channel. Since TLS is designed to operate over a reliable layer, OpenVPN implements an explicit acknowledgement and re-transmission mechanism for its control channel messages [30]. Specifically, incoming P_- Control packets are acknowledged by P_ACK packets, which do not carry any TLS payloads and are uniform in size (Note these ACK packets are carried over by TCP as payload and are not the same as TCP ACK flags). Moreover, these ACK packets are seen mostly only in the early stage of a flow, during the handshake phase, and are not used in the actual data transfer channel, which can run over an unreliable layer. To our knowledge, we are the first to devise fingerprinting attempts based on the distinct protocol-layer ACKs against OpenVPN. Previously, the unique timing pattern in meek’s TCP-level ACK traffic has rendered the obfuscation tool vulnerable to detection [67]. For OpenVPN, the presence of explicit ACK packets, uniform in size and only seen in some parts of a session, provides another fingerprintable feature. Specifically, we first identify a likely ACK packet of a session by locating an initial packet exchange sequence of C- >S (Client-Reset), S->C (Server-Reset), C->S (ACK), C->S (Control), as illustrated in Figure 1. For vanilla OpenVPN and XOR-based obfuscation, the first ACK packet usually appears as the third (data) packet transmitted in a session. For tunnels or obfuscators that have their own handshake or key exchange process (e.g., Stunnel, SSH tunnel, or Obfsproxy), this counting is offset by the number of tunnel handshake packets. Next, we group packets into 10-packet bins, and we derive the ACK fingerprint for each flow by counting the number of packets in each bin that have the same size as the identified ACK packet. For OpenVPN flows, we expect to observe a high number of ACK packets in early bins and an absence of them in later bins. (Later in the session, Control and ACK packets can be exchanged again to transfer random key materials, but it is not expected to be observed within our observation window N.) This approach proves effective to fingerprint vanilla OpenVPN as well as obfuscated services running over encrypted tunnels that lack random padding. We quantify exact fingerprinting thresholds in Section 7.1. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Authors: Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background & Related Work 2 Background & Related Work 3 Challenges in Real-world VPN Detection 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 9.2 Results for all flows 10 Discussion and Mitigations 10 Discussion and Mitigations 11 Conclusion 11 Conclusion 12 Acknowledgement and References 12 Acknowledgement and References Appendix Appendix 6.2 ACK-based Fingerprinting OpenVPN engages in a TLS-style handshake with its peer over the control channel. Since TLS is designed to operate over a reliable layer, OpenVPN implements an explicit acknowledgement and re-transmission mechanism for its control channel messages [30]. Specifically, incoming P_- Control packets are acknowledged by P_ACK packets, which do not carry any TLS payloads and are uniform in size (Note these ACK packets are carried over by TCP as payload and are not the same as TCP ACK flags). Moreover, these ACK packets are seen mostly only in the early stage of a flow, during the handshake phase, and are not used in the actual data transfer channel, which can run over an unreliable layer. To our knowledge, we are the first to devise fingerprinting attempts based on the distinct protocol-layer ACKs against OpenVPN. Previously, the unique timing pattern in meek’s TCP-level ACK traffic has rendered the obfuscation tool vulnerable to detection [67]. For OpenVPN, the presence of explicit ACK packets, uniform in size and only seen in some parts of a session, provides another fingerprintable feature. Specifically, we first identify a likely ACK packet of a session by locating an initial packet exchange sequence of C- >S (Client-Reset), S->C (Server-Reset), C->S (ACK), C->S (Control), as illustrated in Figure 1. For vanilla OpenVPN and XOR-based obfuscation, the first ACK packet usually appears as the third (data) packet transmitted in a session. For tunnels or obfuscators that have their own handshake or key exchange process (e.g., Stunnel, SSH tunnel, or Obfsproxy), this counting is offset by the number of tunnel handshake packets. Next, we group packets into 10-packet bins, and we derive the ACK fingerprint for each flow by counting the number of packets in each bin that have the same size as the identified ACK packet. For OpenVPN flows, we expect to observe a high number of ACK packets in early bins and an absence of them in later bins. (Later in the session, Control and ACK packets can be exchanged again to transfer random key materials, but it is not expected to be observed within our observation window N.) This approach proves effective to fingerprint vanilla OpenVPN as well as obfuscated services running over encrypted tunnels that lack random padding. We quantify exact fingerprinting thresholds in Section 7.1. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Unveiling OpenVPN: Fingerprinting with ACK Patterns and Timing Analysis

For the first post of 2025, I'm continuing my retrospective series. As last year, before diving in into the report proper, I'd like to remind you that Ukraine is still fighting for its survival after nearly 3 years and countless Russian war crimes. The civilized world can't allow rewarding the invasion of a sovereign country, lest we see more invasions. Remember that in 1938, in Munich, European "leaders" acknowledged Hitler's annexation of Czechoslovakia's Sudetenland. It didn't prevent Nazi Germany to invade the rest of Europe a year later; worse, it emboldened it! Please help Ukraine and help the civilized world by doing so. Here are two organizations I trust: Come Back Alive and Dzyga's Paw. I Lost My Job (and Found a New One) First and foremost, I suffered quite a blow this year. It was a complete surprise when my company unexpectedly let me go due to budget reasons. To tell you I was unhappy is quite an understatement. It was also a complete surprise, as my CEO had assured me everything was okay a few months before he notified me. Upon receiving the news on a Thursday morning in mid-August, I didn't let it deter me. I took a one-day break and immediately started my job search on Friday, determined to find a new opportunity. I thought I would get a new job quickly. However, my optimism met reality—it took me 3.5 months to return to work. Here's a couple of stats that may be interesting if you're facing the same issue: I listed 147 job offers over the months; including asking people in my network. I submitted my application to 130 of them. I got 55 answers, counting automated emails. On a couple of submissions, I'm still waiting for the next step. I entered the interview process with 13 companies In the end, I received two offers, one for a consultant role and the other for a Developer Advocate role. After careful consideration, I finally chose the latter, working for Loft Labs. It provides vCluster, a virtualization solution for Kubernetes clusters. Here's my first blog post on vCluster. One additional fun fact regarding the delay in getting answers: It's not uncommon that I received them two months after my application. All in all, it's safe that the current economic situation is bleak. Unfortunately, it impacts IT just as well. Worse, developer advocacy has lost a lot of its appeal as companies are looking for short-term ROI; developer advocates are a long-term investment. We must bite the bullet and hope for better times. I'm Using More and More AI I used AI more than a couple of times this year. Most companies require a cover letter to submit your application. It's completely irrelevant in this day and age, especially in our field. If your HR department insists on wasting my time, don't be irritated because I'm using an IT solution to solve this issue. Pointing to the job description, attaching your CV, and asking your favorite generative AI tool to write the cover letter is dead simple. I had to learn about vCluster and refresh my knowledge of Kubernetes. I used and experimented with generative AI to help me migrate my OpenTelemetry demo to Kubernetes. It was not flawless, but it led me in the correct direction more than once. I used both GitHub Copilot and OpenAI in their paid versions. Compared to last year, I think I got better results with the latter; Copilot is not well integrated with IntelliJ IDEA. I'm an avid learner of many different subjects. So far, my go-to sites were Wikipedia and Wiktionary (for languages). This approach falls short when you have no clue what to search for. I started to rely more and more on OpenAI for this: So far, I'm happy about the results. Finally, I'm not a graphic designer, and I spend a lot of time searching for free illustrations for my blog posts on image banks. I prefer to spend less time on OpenAI to describe what I want. I don't have strong opinions, so it gives me good enough results. Technical Content I published fifty-one blog posts this year on this blog: one weekly on Sunday, but one between Christmas and the New Year for obvious reasons. Here are the top most viewed pages in 2024: Rank

Post

Views

Avg. time on page



#1

A list of cache providers

5,901

1:02



#2

OpenTelemetry Tracing on Spring Boot, Java Agent vs. Micrometer Tracing

5,845

1:03



#3

Using my new Raspberry Pi to run an existing GitHub Action

4,985

:45



#4

Multiple Spring Boot applications in the same project

4,421

:40



#5

My final take on Gradle (vs. Maven)

4,132

1:59 I continue to cross-post on different sites. Here is my followers' count on the sites that provide metrics: Site

2024

2023

2022



Medium

1,031

741

564



Dev.to

17,126

8,156

1,838



Hashnode

104

89

80 The trend I noticed last year is confirmed: wild growth on dev.to and close to plateau on Hashnode. Comparing Social Media My metrics tracking script is still working nicely. Do I need to mention the crazy Bluesky growth? Note that I moved my handle from @nfrankel.bsky.social to @frankel.ch. On the other hand, numbers on Twitter (I won't call it X) are dwindling, to say the least. Worse, after the US presidential elections, I noticed that I lost hundreds of followers; the count went back to around what it was in the summer. Even worse, the number of followers doesn't translate into interactions. For example, I've published the same content on Twitter, Bluesky, Mastodon, and LinkedIn. I'll leave Mastodon out since it doesn't offer metrics on the content. Here they are: https://x.com/nicolas_frankel/status/1874143229198696786?embedable=true https://bsky.app/profile/frankel.ch/post/3lemmp3vffk2y?embedable=true https://www.linkedin.com/embed/feed/update/urn:li:share:7279804263545925632?embedable=true We can't compare LinkedIn since it's mostly about people you know personally; there's a higher social incentive to interact with others. When I wrote this post, I had slightly more likes on Twitter (13) than on Bluesky (11), but with more than ten times the followers on the former. Twitter is going down, and that's without even mentioning all the alerts that it has become a massive vector of hate and disinformation. I recommend keeping your account if you already have one there, but new investments are a total loss of both time and money. Summary My goals are evident this year. I've joined a new company, so I need to learn more about the product, create new related content, design new talks, submit them, and be selected. In the meantime, feel free to invite me to your meetup and user group. Happy New Year! Originally published at A Java Geek on January 5th, 2025 For the first post of 2025, I'm continuing my retrospective series. As last year, before diving in into the report proper, I'd like to remind you that Ukraine is still fighting for its survival after nearly 3 years and countless Russian war crimes. The civilized world can't allow rewarding the invasion of a sovereign country, lest we see more invasions. Remember that in 1938, in Munich, European "leaders" acknowledged Hitler's annexation of Czechoslovakia's Sudetenland. It didn't prevent Nazi Germany to invade the rest of Europe a year later; worse, it emboldened it! Please help Ukraine and help the civilized world by doing so. Here are two organizations I trust: Come Back Alive and Dzyga's Paw . Come Back Alive Dzyga's Paw I Lost My Job (and Found a New One) First and foremost, I suffered quite a blow this year. It was a complete surprise when my company unexpectedly let me go due to budget reasons. To tell you I was unhappy is quite an understatement. It was also a complete surprise, as my CEO had assured me everything was okay a few months before he notified me. Upon receiving the news on a Thursday morning in mid-August, I didn't let it deter me. I took a one-day break and immediately started my job search on Friday, determined to find a new opportunity. I thought I would get a new job quickly. However, my optimism met reality—it took me 3.5 months to return to work. Here's a couple of stats that may be interesting if you're facing the same issue: I listed 147 job offers over the months; including asking people in my network. I listed 147 job offers over the months; including asking people in my network. I submitted my application to 130 of them. I submitted my application to 130 of them. I got 55 answers, counting automated emails. I got 55 answers, counting automated emails. On a couple of submissions, I'm still waiting for the next step. On a couple of submissions, I'm still waiting for the next step. I entered the interview process with 13 companies I entered the interview process with 13 companies In the end, I received two offers, one for a consultant role and the other for a Developer Advocate role. After careful consideration, I finally chose the latter, working for Loft Labs. It provides vCluster, a virtualization solution for Kubernetes clusters. Here's my first blog post on vCluster. In the end, I received two offers, one for a consultant role and the other for a Developer Advocate role. After careful consideration, I finally chose the latter, working for Loft Labs . It provides vCluster , a virtualization solution for Kubernetes clusters. Here's my first blog post on vCluster . Loft Labs vCluster blog post on vCluster One additional fun fact regarding the delay in getting answers: It's not uncommon that I received them two months after my application. All in all, it's safe that the current economic situation is bleak. Unfortunately, it impacts IT just as well. Worse, developer advocacy has lost a lot of its appeal as companies are looking for short-term ROI; developer advocates are a long-term investment. We must bite the bullet and hope for better times. I'm Using More and More AI I used AI more than a couple of times this year. Most companies require a cover letter to submit your application. It's completely irrelevant in this day and age, especially in our field. If your HR department insists on wasting my time, don't be irritated because I'm using an IT solution to solve this issue. Pointing to the job description, attaching your CV, and asking your favorite generative AI tool to write the cover letter is dead simple. I had to learn about vCluster and refresh my knowledge of Kubernetes. I used and experimented with generative AI to help me migrate my OpenTelemetry demo to Kubernetes. It was not flawless, but it led me in the correct direction more than once. I used both GitHub Copilot and OpenAI in their paid versions. Compared to last year, I think I got better results with the latter; Copilot is not well integrated with IntelliJ IDEA. OpenTelemetry demo I'm an avid learner of many different subjects. So far, my go-to sites were Wikipedia and Wiktionary (for languages). This approach falls short when you have no clue what to search for. I started to rely more and more on OpenAI for this: So far, I'm happy about the results. Finally, I'm not a graphic designer, and I spend a lot of time searching for free illustrations for my blog posts on image banks. I prefer to spend less time on OpenAI to describe what I want. I don't have strong opinions, so it gives me good enough results. Technical Content I published fifty-one blog posts this year on this blog : one weekly on Sunday, but one between Christmas and the New Year for obvious reasons. Here are the top most viewed pages in 2024: this blog Rank

Post

Views

Avg. time on page



#1

A list of cache providers

5,901

1:02



#2

OpenTelemetry Tracing on Spring Boot, Java Agent vs. Micrometer Tracing

5,845

1:03



#3

Using my new Raspberry Pi to run an existing GitHub Action

4,985

:45



#4

Multiple Spring Boot applications in the same project

4,421

:40



#5

My final take on Gradle (vs. Maven)

4,132

1:59 Rank

Post

Views

Avg. time on page #1

A list of cache providers

5,901

1:02 #2

OpenTelemetry Tracing on Spring Boot, Java Agent vs. Micrometer Tracing

5,845

1:03 #3

Using my new Raspberry Pi to run an existing GitHub Action

4,985

:45 #4

Multiple Spring Boot applications in the same project

4,421

:40 #5

My final take on Gradle (vs. Maven)

4,132

1:59 Rank Post Views Avg. time on page Rank Rank Post Post Views Views Avg. time on page Avg. time on page #1 A list of cache providers 5,901 1:02 #1 #1 A list of cache providers A list of cache providers A list of cache providers 5,901 5,901 1:02 1:02 #2 OpenTelemetry Tracing on Spring Boot, Java Agent vs. Micrometer Tracing 5,845 1:03 #2 #2 OpenTelemetry Tracing on Spring Boot, Java Agent vs. Micrometer Tracing OpenTelemetry Tracing on Spring Boot, Java Agent vs. Micrometer Tracing OpenTelemetry Tracing on Spring Boot, Java Agent vs. Micrometer Tracing 5,845 5,845 1:03 1:03 #3 Using my new Raspberry Pi to run an existing GitHub Action 4,985 :45 #3 #3 Using my new Raspberry Pi to run an existing GitHub Action Using my new Raspberry Pi to run an existing GitHub Action Using my new Raspberry Pi to run an existing GitHub Action 4,985 4,985 :45 :45 #4 Multiple Spring Boot applications in the same project 4,421 :40 #4 #4 Multiple Spring Boot applications in the same project Multiple Spring Boot applications in the same project Multiple Spring Boot applications in the same project 4,421 4,421 :40 :40 #5 My final take on Gradle (vs. Maven) 4,132 1:59 #5 #5 My final take on Gradle (vs. Maven) My final take on Gradle (vs. Maven) My final take on Gradle (vs. Maven) 4,132 4,132 1:59 1:59 I continue to cross-post on different sites. Here is my followers' count on the sites that provide metrics: Site

2024

2023

2022



Medium

1,031

741

564



Dev.to

17,126

8,156

1,838



Hashnode

104

89

80 Site

2024

2023

2022 Medium

1,031

741

564 Dev.to

17,126

8,156

1,838 Hashnode

104

89

80 Site 2024 2023 2022 Site Site 2024 2024 2023 2023 2022 2022 Medium 1,031 741 564 Medium Medium Medium 1,031 1,031 741 741 564 564 Dev.to 17,126 8,156 1,838 Dev.to Dev.to Dev.to 17,126 17,126 8,156 8,156 1,838 1,838 Hashnode 104 89 80 Hashnode Hashnode Hashnode 104 104 89 89 80 80 The trend I noticed last year is confirmed: wild growth on dev.to and close to plateau on Hashnode. dev.to Comparing Social Media My metrics tracking script is still working nicely. Do I need to mention the crazy Bluesky growth? Note that I moved my handle from @nfrankel.bsky.social to @frankel.ch . Bluesky @nfrankel.bsky.social @frankel.ch On the other hand, numbers on Twitter (I won't call it X) are dwindling, to say the least. Worse, after the US presidential elections, I noticed that I lost hundreds of followers; the count went back to around what it was in the summer. Even worse, the number of followers doesn't translate into interactions. For example, I've published the same content on Twitter, Bluesky, Mastodon, and LinkedIn. I'll leave Mastodon out since it doesn't offer metrics on the content. Here they are: https://x.com/nicolas_frankel/status/1874143229198696786?embedable=true https://x.com/nicolas_frankel/status/1874143229198696786?embedable=true https://bsky.app/profile/frankel.ch/post/3lemmp3vffk2y?embedable=true https://bsky.app/profile/frankel.ch/post/3lemmp3vffk2y?embedable=true https://www.linkedin.com/embed/feed/update/urn:li:share:7279804263545925632?embedable=true https://www.linkedin.com/embed/feed/update/urn:li:share:7279804263545925632?embedable=true We can't compare LinkedIn since it's mostly about people you know personally; there's a higher social incentive to interact with others. When I wrote this post, I had slightly more likes on Twitter (13) than on Bluesky (11), but with more than ten times the followers on the former. Twitter is going down, and that's without even mentioning all the alerts that it has become a massive vector of hate and disinformation. I recommend keeping your account if you already have one there, but new investments are a total loss of both time and money. Summary My goals are evident this year. I've joined a new company, so I need to learn more about the product, create new related content, design new talks, submit them, and be selected. In the meantime, feel free to invite me to your meetup and user group. Happy New Year! Originally published at A Java Geek on January 5th, 2025 Originally published at A Java Geek on January 5th, 2025 A Java Geek

2024: A Retrospective

Case Study: Evaluating GNN Performance Using Shortest-Path Distance for Generalization and Fairness

Will Artificial Intelligence Take Over Humans?

አርቴፊሻል ኢንተለጀንስ ሰዎችን ይቆጣጠራሉ?

¿La Inteligencia Artificial superará a los humanos?

L'intelligenza artificiale prenderà il sopravvento sugli esseri umani?

人工知能は人間を支配するのか？

Жасанды интеллект адамдарды жаулап ала ма?

Mayele ya mayele ekozwa bato?

¿Inteligencia Artificialchu Runakunata hap’ikapunqa?

Va prelua inteligența artificială pe oameni?

Оё зеҳни сунъӣ одамонро тасарруф мекунад?

ปัญญาประดิษฐ์จะเข้ามาแทนที่มนุษย์ได้หรือไม่?

Maaagaw ba ng Artipisyal na Katalinuhan ang mga Tao?

人工智能会取代人类吗？

To Interact Better, Interact More for Fun

Authors:
(1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co);
(2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team;
(3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team;
(14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction
Related Work
Method
Experimental Details
Results and Ablations
Conclusions and Limitations , Acknowledgements and References
Appendix 4 EXPERIMENTAL DETAILS We conduct all of our fine-tuning experiments using Mistral 7B (Jiang et al., 2023), which is the current state-of-the-art base LM at the 7B parameter scale, and matches the performance of much larger models like LLaMa 34B on many NLP benchmarks. We use the Transformer Reinforcement Learning (TRL) library for fine-tuning (von Werra et al., 2020), in conjunction with DeepSpeed ZeRO3 (Rajbhandari et al., 2020) and FlashAttention-2 (Dao, 2023) to optimize memory and improve training speed. All models are trained with the AdamW optimizer and no weight decay. We did not experiment with parameter-efficient techniques such as LoRA (Hu et al., 2021), but expect similar results to hold with these methods. All experiments were run on 16 A100s using bfloat16 precision and typically took 2-4 hours to complete. For the full set of hyperparameters and instructions on how to train the models, see: https://github.com/huggingface/alignment-handbook. 4.1 DATASETS We focus on two dialogue datasets that have been distilled from a mix of open and proprietary models, and have previously been shown to produce strong chat models like the UltraLM (Ding et al., 2023): • UltraChat (Ding et al., 2023) is a self-refinement dataset consisting of 1.47M multi-turn dialogues generated by GPT-3.5-TURBO over 30 topics and 20 different types of text material. We initially ran dSFT over the whole corpus, but found the resulting chat model had a tendency to respond with incorrect capitalization and would preface its answers with phrases such as “I don’t have personal experiences”, even for straightforward questions like “How do I clean my car?”. To handle these issues in the training data, we applied truecasing heuristics to fix the grammatical errors (approximately 5% of the dataset), as well as several filters to focus on helpfulness and remove the undesired model responses. The resulting dataset contains approximately 200k examples. • UltraFeedback (Cui et al., 2023) consists of 64k prompts, each of which have four LLM responses that are rated by GPT-4 according to criteria like instruction-following, honesty, and helpfulness. We construct binary preferences from UltraFeedback by selecting the highest mean score as the “chosen” response and one of the remaining three at random as “rejected”. We opted for random selection instead of selecting the lowest-scored response to encourage diversity and make the DPO objective more challenging. As noted above, this step is computed offline and does not involve any sampling from the reference model. We make the pre-processed datasets available on the Hugging Face Hub.1 4.2 EVALUATION Our main evaluations are on single-turn and multi-turn chat benchmarks that measure a model’s ability to follow instructions and respond to challenging prompts across a diverse range of domains: • MT-Bench (Zheng et al., 2023) is a multi-turn benchmark that consists of 160 questions across eight different areas of knowledge. In this benchmark, the model must answer an initial question, and then provide a second response to a predefined followup question. Each model response is then rated by GPT-4 on a scale from 1-10, with the final score given by the mean over the two turns. • AlpacaEval (Li et al., 2023) is a single-turn benchmark where a model must generate a response to 805 questions on different topics, mostly focused on helpfulness. Models are also scored by GPT-4, but the final metric is the pairwise win-rate against a baseline model (text-davinci-003). We also evaluate ZEPHYR-7B on the Open LLM Leaderboard (Beeching et al., 2023), which measures the performance of LMs across four multiclass classification tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), and Truthful QA(Lin et al., 2022). Although this leaderboard does not directly measure the conversational quality of chat models, it does provide a useful signal to validate whether fine-tuning has introduced regressions on the base model’s reasoning and truthfulness capabilities. Across all benchmarks, we compare ZEPHYR-7B against a variety of open and proprietary models, each with different alignment procedures. To facilitate comparison across open model sizes, we group our comparisons in terms of 7B models (XWIN-LM (Team, 2023), MISTRALINSTRUCT (Jiang et al., 2023), MPT-CHAT (ML, 2023), and STABLELM-α), as well as larger models up to 70B parameters (LLAMA2-CHAT (Touvron et al., 2023), VICUNA˜ (Chiang et al., 2023), WizardLM (Xu et al.), and GUANACO (Dettmers et al., 2023)). For the chat benchmarks, we also compare against proprietary models, including CLAUDE 2, GPT-3.5-TURBO and GPT-4 (OpenAI, 2023). 4.3 DETAILS OF SFT TRAINING We train our SFT models for one to three epochs. We use a cosine learning rate scheduler with a peak learning rate of 2e-5 and 10% warmup steps. We train all models with a global batch size of 512 and use packing with a sequence length of 2048 tokens. 4.4 DETAILS OF DPO TRAINING Similar to SFT, we train our DPO models for one to three epochs. We use a linear learning rate scheduler with a peak learning rate of 5e-7 and 10% warmup steps. We train all models with a global batch size of 32 and use β = 0.1 from Eq. (1) to control the deviation from the reference model. The final ZEPHYR-7B model was initialized from the SFT model that was trained for one epoch and further optimized for three DPO epochs (see Figure 3 for an epoch ablation on MT-Bench). This paper is available on arxiv under CC 4.0 license. 1 https://huggingface.co/collections/HuggingFaceH4/ zephyr-7b-6538c6d6d5ddd1cbb1744a66 Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Authors: Authors: (1) Lewis Tunstall,  Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team (email: lewis@huggingface.co); (2) Edward Beeching, Equal contribution and The H4 (Helpful, Honest, Harmless, Huggy) Team; (3) Nathan Lambert, The H4 (Helpful, Honest, Harmless, Huggy) Team; (4) Nazneen Rajani, The H4 (Helpful, Honest, Harmless, Huggy) Team; (5) Kashif Rasul, The H4 (Helpful, Honest, Harmless, Huggy) Team; (6) Younes Belkada, The H4 (Helpful, Honest, Harmless, Huggy) Team; (7) Shengyi Huang, The H4 (Helpful, Honest, Harmless, Huggy) Team; (8) Leandro von Werra, The H4 (Helpful, Honest, Harmless, Huggy) Team; (9) Clementine Fourrier, The H4 (Helpful, Honest, Harmless, Huggy) Team; (10) Nathan Habib, The H4 (Helpful, Honest, Harmless, Huggy) Team; (11) Nathan Sarrazin, The H4 (Helpful, Honest, Harmless, Huggy) Team; (12) Omar Sanseviero, The H4 (Helpful, Honest, Harmless, Huggy) Team; (13) Alexander M. Rush, The H4 (Helpful, Honest, Harmless, Huggy) Team; (14) Thomas Wolf, The H4 (Helpful, Honest, Harmless, Huggy) Team. Table of Links Abstract and Introduction Related Work Method Experimental Details Results and Ablations Conclusions and Limitations , Acknowledgements and References Appendix Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experimental Details Experimental Details Results and Ablations Results and Ablations Conclusions and Limitations , Acknowledgements and References Conclusions and Limitations , Acknowledgements and References Appendix Appendix 4 EXPERIMENTAL DETAILS We conduct all of our fine-tuning experiments using Mistral 7B (Jiang et al., 2023), which is the current state-of-the-art base LM at the 7B parameter scale, and matches the performance of much larger models like LLaMa 34B on many NLP benchmarks. We use the Transformer Reinforcement Learning (TRL) library for fine-tuning (von Werra et al., 2020), in conjunction with DeepSpeed ZeRO3 (Rajbhandari et al., 2020) and FlashAttention-2 (Dao, 2023) to optimize memory and improve training speed. All models are trained with the AdamW optimizer and no weight decay. We did not experiment with parameter-efficient techniques such as LoRA (Hu et al., 2021), but expect similar results to hold with these methods. All experiments were run on 16 A100s using bfloat16 precision and typically took 2-4 hours to complete. For the full set of hyperparameters and instructions on how to train the models, see: https://github.com/huggingface/alignment-handbook. 4.1 DATASETS We focus on two dialogue datasets that have been distilled from a mix of open and proprietary models, and have previously been shown to produce strong chat models like the UltraLM (Ding et al., 2023): • UltraChat (Ding et al., 2023) is a self-refinement dataset consisting of 1.47M multi-turn dialogues generated by GPT-3.5-TURBO over 30 topics and 20 different types of text material. We initially ran dSFT over the whole corpus, but found the resulting chat model had a tendency to respond with incorrect capitalization and would preface its answers with phrases such as “I don’t have personal experiences”, even for straightforward questions like “How do I clean my car?”. To handle these issues in the training data, we applied truecasing heuristics to fix the grammatical errors (approximately 5% of the dataset), as well as several filters to focus on helpfulness and remove the undesired model responses. The resulting dataset contains approximately 200k examples. • UltraChat • UltraFeedback (Cui et al., 2023) consists of 64k prompts, each of which have four LLM responses that are rated by GPT-4 according to criteria like instruction-following, honesty, and helpfulness. We construct binary preferences from UltraFeedback by selecting the highest mean score as the “chosen” response and one of the remaining three at random as “rejected”. We opted for random selection instead of selecting the lowest-scored response to encourage diversity and make the DPO objective more challenging. As noted above, this step is computed offline and does not involve any sampling from the reference model. • UltraFeedback We make the pre-processed datasets available on the Hugging Face Hub.1 4.2 EVALUATION Our main evaluations are on single-turn and multi-turn chat benchmarks that measure a model’s ability to follow instructions and respond to challenging prompts across a diverse range of domains: • MT-Bench (Zheng et al., 2023) is a multi-turn benchmark that consists of 160 questions across eight different areas of knowledge. In this benchmark, the model must answer an initial question, and then provide a second response to a predefined followup question. Each model response is then rated by GPT-4 on a scale from 1-10, with the final score given by the mean over the two turns. • MT-Bench • AlpacaEval (Li et al., 2023) is a single-turn benchmark where a model must generate a response to 805 questions on different topics, mostly focused on helpfulness. Models are also scored by GPT-4, but the final metric is the pairwise win-rate against a baseline model (text-davinci-003). • AlpacaEval We also evaluate ZEPHYR-7B on the Open LLM Leaderboard (Beeching et al., 2023), which measures the performance of LMs across four multiclass classification tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), and Truthful QA(Lin et al., 2022). Although this leaderboard does not directly measure the conversational quality of chat models, it does provide a useful signal to validate whether fine-tuning has introduced regressions on the base model’s reasoning and truthfulness capabilities. Across all benchmarks, we compare ZEPHYR-7B against a variety of open and proprietary models, each with different alignment procedures. To facilitate comparison across open model sizes, we group our comparisons in terms of 7B models (XWIN-LM (Team, 2023), MISTRALINSTRUCT (Jiang et al., 2023), MPT-CHAT (ML, 2023), and STABLELM-α), as well as larger models up to 70B parameters (LLAMA2-CHAT (Touvron et al., 2023), VICUNA˜ (Chiang et al., 2023), WizardLM (Xu et al.), and GUANACO (Dettmers et al., 2023)). For the chat benchmarks, we also compare against proprietary models, including CLAUDE 2, GPT-3.5-TURBO and GPT-4 (OpenAI, 2023). 4.3 DETAILS OF SFT TRAINING We train our SFT models for one to three epochs. We use a cosine learning rate scheduler with a peak learning rate of 2e-5 and 10% warmup steps. We train all models with a global batch size of 512 and use packing with a sequence length of 2048 tokens. 4.4 DETAILS OF DPO TRAINING Similar to SFT, we train our DPO models for one to three epochs. We use a linear learning rate scheduler with a peak learning rate of 5e-7 and 10% warmup steps. We train all models with a global batch size of 32 and use β = 0.1 from Eq. (1) to control the deviation from the reference model. The final ZEPHYR-7B model was initialized from the SFT model that was trained for one epoch and further optimized for three DPO epochs (see Figure 3 for an epoch ablation on MT-Bench). This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv 1 https://huggingface.co/collections/HuggingFaceH4/ zephyr-7b-6538c6d6d5ddd1cbb1744a66

Stories, News & Mentions about WEGROUP

Categories

Trending Topics

Stories, News & Mentions about WEGROUP

Wegroup’s Stories on HackerNoon

Wegroup's News & Mentions

Categories

Trending Topics