廢話不多說，直接乾貨。

一、oracle去重

1、創建測試數據

create table test_duplicate_removal(
       c001 number,
       c002 varchar2(100)
);
insert into test_duplicate_removal values(101, 'aa');
insert into test_duplicate_removal values(102, 'aa');
insert into test_duplicate_removal values(103, 'aa');
insert into test_duplicate_removal values(104, 'bb');
insert into test_duplicate_removal values(105, 'bb');
insert into test_duplicate_removal values(106, 'cc');
insert into test_duplicate_removal values(107, 'cc');
insert into test_duplicate_removal values(108, 'dd');

2、使用row_number() over()函數根據C002列去重

創建一個rn列，根據C002進行分組，每個小組內再根據C001的值進行排序。

select c001,c002, row_number()  over(partition by c002 order by c001 desc) rn from  test_duplicate_removal

通過rn篩選值為1的行，同時也就對C002進行了去重

select * from (select c001,c002, row_number()  over(partition by c002 order by c001 desc) rn from  test_duplicate_removal) t where t.rn=1

二、python的pandas模塊去重方法

1、將數據庫數據導出保存為CSV

2、pandas實現sql里排序函數row_number() over()功能

import pandas as pd
# 讀取CSV數據
df = pd.read_csv('test_duplicate_removal.csv')
print('打印原始數據：')
print(df)
# 此處等價於sql里的排序函數row_number() over()功能
df['RN'] = df['C001'].groupby(df['C002']).rank()
print()
print('根據C002分組，根據C001組內排序：')
print(df)
# 去重
print()
print('去重，篩選RN=1的行：')
print(df[df['RN'] == 1])

運行結果